The Dilemma that is WordPress Security
The past few weeks WordPress Security has come to the forefront of the discussion again, as it often does every few months. As is often the case, it’s highly emotional and generates a lot of discussion. Chris Lema shared a post, Our discussions around WordPress security should change, and that sparked some interesting conversations.
He’s absolutely right, it should.
What many fail to realize within the community however is that the crux of the problem goes beyond Access Control and Software Vulnerabilities. The problem is the end-user, the website owner, the grandma turned website owner, the full-time mommy turned SEO expert, and the message that is pushed from top down through the various niches / factions of clicks within the community.
The irony of it all is that it revolves around the concept that made WordPress so popular — it’s ease of use.
But First, A Story… That Seems to be the thing…
I was a United States Marine at one point in my not so long ago past. I spent close to 1.5 years in Iraq over a 2 year period. At one point I was the Geospatial Intelligence lead for a 3 provinces. I would need a series of posts to explain what that is, but the gist of it is I analyzed geography to understand the implication of specific actions. Can we go that route? If we do, what happens. What are the potential pitfalls, etc…
The standard classification in the space when constructing things like Cross Country Mobility Studies was always Red, Yellow, Green. Red always meant bad, Yellow meant caution. Green met good. Pretty standard I’d say, something that we all understand as humans. So standard it’s something we deal with every day at stop lights around the world.
One day under a series of combat operations. Everything was a buzz in the command center. We had operations going. Had various high value targets in mind. One specific event was to occur in 24 hours. This event however require cross country mobility (i.e., no roads or infrastructure to be leveraged). I worked with our various analysts (i.e., weather, all source intel, open source) etc.. to understand what the environment would be like. Based on the composition of the soil and vegetation, coupled with the locals told of us of the area, bundled with the state of the weather, the state of the two main axis of advance were going to be no-go’s.
As was the approach, I prepared my report. Prepared my presentation. And presented it to the commander of the operation with a well thought out and prepared plan, clearly articulating the analysis and the implications of why we could not do those approaches (i.e., the attack) at that time.
Commanders response: “What is all that red stuff on the map?”
My response: “Sir, that is the no-go area.”
Commander response: “But that’s where we need to go.”
My response: “I understand sir, but that’s a no-go, it must be red to signify we can’t go that route.”
Commander response: “That’s not going to work. Need you to change that to something less drastic before I send to the guys on the ground.”
I did as I was told. The mission failed. Every vehicle that went through the entry point got stuck do to inclement weather and the impact that had on the terrain. Was it because of me? I have no idea, but what if I would have followed the established protocol? Would they have seen the issues and delayed the operation?
The WordPress Security Challenge is Communication
WordPress and Security and the Dilemma that Ensues
When I look at the challenges with WordPress Security I boil them down to few things for me:
1: The communities insatiable appetite for debate.
To be fair though, this is more the world of social media and open communication than WordPress specifically. There is however an interesting dynamic that I have noticed creep into the WordPress ecosystem over the past few years and as is grows it’s grows exponentially as well. It’s the social mob mentality which turns highly irrational and emotional – I too am victim to this.
In security specifically it is rampant.
Here is the thing to note. Personally, I love intellectual debate, it’s healthy, it should be encouraged. The problem we have in WordPress security is that we’re not having intellectual debates. Everyone has an opinion for the sake of having an opinion with little to no understanding of the domain.
2: Security is categorized through the lens of the platform and not security.
What this talks to is specifically platform bias, so much so that it clouds judgement. The community spends more time defending a weakness than acknowledging it and moving on. This is a hard lesson that others learned long ago, just look at Microsoft.
This unfortunately is brought about because of emotion and that emotion is high, because in relative terms, this community is very young and immature. I don’t say that in a negative sense, I mean that sincerely. It is.
I had a great conversation with a friend of mine that said:
Executives of companies get in trouble when they go off-script. In WordPress however, it’s a bunch of adolescents, and I don’t argue with 12 year olds.
I found that to be very profound insight. The beauty is, it wasn’t meant derogatory, but it struck at the issue that the community has a whole, far beyond Security.
In the past few months I have heard from several people, in passing and in presentations:
Gosh how I hate presentations that scare end-users.
or
Many talks out there will tell you of all these security issues in WordPress, but I’m hear to put a lighter spin on the discussion. To show you it’s not a scary place.
I’ll be honest, I was a bit dumbfounded by this. This idea that if we call it something nicer for the sake of calling it nicer is preposterous to me. Do allow me to caveat, I’m obviously not saying we should call a puppy a lion, well, unless it’s a lion pup.
An example of what I mean can be found this last week when we look at the recent Security patch pushed by the Core team. The key however is in their message to the community:
This release fixes a possible denial of service issue in PHP’s XML processing, reported by Nir Goldshlager of the Salesforce.com Product Security Team
This in turn leads to a butterfly and flowers post about collaboration, which was great, until you read:
The bug itself is relatively minor, but of interest is the collaboration between the WordPress and Drupal teams to create a fix.
You then look at Drupal’s release:
The PHP XML parser used by this XML-RPC endpoint is vulnerable to an XML entity expansion attack and other related XML payload attacks which can cause CPU and memory exhaustion and the site’s database to reach the maximum number of open connections. Any of these may lead to the site becoming unavailable or unresponsive (denial of service).
They even provide the correct classification:
Security risk: 13/25 ( Moderately Critical) AC:None/A:None/CI:None/II:None/E:Proof/TD:100
In which they explain, objectively what that categorization means:
Remotely exploitable vulnerabilities that can compromise the system. Interaction (such as an administrator viewing a particular page) is required for this exploit to be successful. Exploits have not yet occurred on systems when vulnerability was disclosed. The exploit requires the user to be registered at the site and have some non-default permission, such as creating content.
Previous examples include: Cross Site Scripting, Access bypass.
Before we turn this into a, off with Tony’s head. Let’s put it on a spike. He says WordPress sucks. Ban him from all WordCamps. I hate his company. Neither of which I have stated, but will undoubtedly be interrupted.
Let’s take a minute to objectively think about that. Drupal was correct, WordPress was not.
Yet, I had a conversation with a developer in which it was explained to me why the severity of this was minor:
Submitting massive POST requests isn’t really any different.
And here in lies the problem. We are so clouded by our greatness that it leads us down the wrong path when it comes to security. Every other platform and technology has learned how to deal with security issues, yet as a WordPress community we have not.
3: Everyone is an Expert
I can think back to 2010, a time where there were more issues in Core than anyone has seen in years. It was the year of big breaches at places like GoDaddy and many other hosts. It’s actually where and when all the discussion of how bad shared hosts firs started. It is also when the recommendation of the prefix started. What many don’t realize is that the reason the prefix even started was because of the shared hosts and the way they configured the environments. The attackers had written a script just looking for the prefix and in doing so quickly attacked multiple servers. It was actually not because of remote attacks, but internal ones. Awkward, I know.
This led us into 2011 where we saw some of the biggest issues around security specifically with a little script called TimThumb. Oh what a cluster that was.
Everyone scurrying, everyone trying to understand what had happened.
Oh and how in 4 years everything has changed. Today, everyone knows. Everyone knows that if you want to stay safe you must:
- Change the Admin user
- You must update
- You must not use the default DB prefix
- You must use SFTP, not FTP
- You better have SSL installed.
- Oh and so many more gems….
This followed by statements like:
The key to security is updates, not auditing
Matt Mullenwegg
These do nothing more than amplify the issue we have loud and clear, it amplifies are lack of understanding of the security space. These are things we have heard, yet have very little understanding for. It is not to say that some don’t have some value, some do, absolutely. The reality however is that attacks and payload’s have become ever so clever. The ones that cause the most damage to the communities actually leverage very little of the things everyone just loves to recommend.
The internet has made us all experts about everything though. We are security experts. We are marketing experts. We are business experts. We are SEO experts. We are WordPress experts. The harsh reality is, we’re experts at search, not knowledge. As a community, this is forgotten and that leads to this overwhelming number of ridiculous posts that carry little merit and provide very little value to the end-user. In reality they cause confusion and deteriorates the end-users trust in the platform.
This additional noise makes it difficult for real information to find it’s way to the surface.
The Fix is Easier Than Most Realize
Unlike Chris’ email my recommendation is not partnerships, but better communication. Openness and disclosure, collaboration if required, and the value that comes from learning from our predecessors. We should be looking at technologies like Java and Windows, and organizations like Microsoft and Oracle for guidance on how security should be handled.
Remember, security is not an open-source thing. It’s not a WordPress thing. Security has been around a lot longer than most technologies. There are established principles and practices in place, defined over decades of experience. While a platform like WordPress is awesome, it is not the end all to be all for all facets of life, and we have to bear that in mind. It is possible that insight comes from other domains that are older and more mature than this one.
I leave that with a very insight post by Jason Cohen, in which he talks to the challenges of Growth. While the post is not specific to security, he said one thing that holds true to us in WordPress Security:
The hardest thing about growth is that rare things become common.
Bear that in mind. You can’t own 22% of the market and not expect issues. Is it best to fight it and blindly convince yourself that they don’t exist, or is it better to embrace it?
Tony, can you clarify further on the statement that you make here?
“Everyone scurrying, everyone trying to understand what had happened.
Oh and how in 4 years everything has changed. Today, everyone knows. Everyone knows that if you want to stay safe you must:
Change the Admin user
You must update
You must not use the default DB prefix
You must use SFTP, not FTP
You better have SSL installed.
Oh and so many more gems…”
I don’t really understand what you are saying here. Because you follow up with this:
“…These do nothing more than amplify the issue we have loud and clear, it amplifies are lack of understanding of the security space.”
How does it amplify the issue? What should we be doing then?
Hi Wayne
Yes, I’ll provide a follow up post specifically for end-users. This post was not for end-users. :)
Thanks
Cool! Thanks brother. Semper Fi
I think you’ve nailed the issue when you point to the fact that WordPress has to maintain its reputation for being so easy to use: “The problem is the end-user, the website owner, the grandma turned website owner, the full-time mommy turned SEO expert…..”
There are many people financially invested in maintaining this perception. God forbid we actually convey to people that owning a website is a responsibility! Great thought-provoking post!
Hey Lucy
Been a while. Yes, you’re right. We like to throw comments around about what is or isn’t better for the community, but that is driven by economic gain.
WordPress is not facing something new. We’ve seen it in every industry. That aspect of it though is not my interest here.
This piece is however:
That is exactly the point. We have to do a better of job, as a community, to convey to the end-user the reality of the platform. In a realistic manner, not in a the rose covered glasses approach. Oh, it’s not that bad.
It reminds me of a conversation I had with a prominent WP dev where I explained that Brute Force attacks were happening at large. The response I got was, yeah that’s impossible because of network latency etc.. This was close to two years ago. Now what is the subject everyone talks about?
Tony
Hey Tony,
GEOINT I have a good friend that works here http://www.3g-consult.de/
This is Australian site below but might give people some insight into the job in the Marine Corps however I could be completely off.
http://www.defence.gov.au/digo/geoint.htm
I like the quote here underneath that have added one.
“I had a great conversation with a friend of mine that said:
Executives of companies get in trouble when they go off-script. In WordPress however, it’s a bunch of adolescents, and I don’t argue with 12 year olds.”
This quote you mentioned above. Reminds me of something my father would tell me.
“Never argue with someone you have to educate first” it has come in I extremely handy at times.
It is ignorant not try and learn from what has come before. When he spoke of Microsoft and Java/Oracle of course we should be taking in everything we can from every possible source on this subject.
Security is universal.
great post,
Tom
Thanks Tom
Love that reference, and impressed you’re familiar with GEOINT. Very cool.
I’ll be putting out a separate post to better articulate my point about Microsoft / Oracle, the point seems to be missed on a few.
Tony
I have a love affair with WordPress. It makes my life so much easier, so it’s easy to look at it with rose colored glasses. At the same time, that isn’t doing the community or the security of our websites any favors. I’m not a fan of Drupal but how they classify their security releases is a system that would be awesome if WordPress adopted. And objectively looking at how other businesses handle security vs. how WordPress does is something we should be doing to help make WordPress better.
I have to say Tony, I really respect your openness not only on security but on how as a community we could improve. In our rush to help others with their security, we need to realize we’re not the security experts and to make sure to fully educate ourselves before speaking on the subject. Looking forward to more blog posts on the subject!
Hi Tish
Thanks so much for stopping by and sharing your thoughts. I believe your statements to be accurate. I feel it’s ok to love something greatly, and yet be realistic in terms of it’s weaknesses and strengths. That’s really the one thing I was trying to convey.
When we can stop convincing ourselves that there isn’t a problem, that, that is when we’ll start making real progress. As of yet though, we’re more in defensive, protect the kingdom, mode.
Tony
I’m curious as to why both of the “problem” users were female, namely the “the grandma turned website owner, the full-time mommy turned SEO expert.” Were you quoting someone else? I find it troublesome that it characterizes women as not knowing what they’re doing with WordPress. One of the things I like most about the WordPress community is that there are so many competent and talented women involved on so many levels. When I go to a Wordcamp conference the line to the women’s restroom is about as long as the line to the men’s restroom, which is a rarity at tech conferences.
Hi Jennette
I’ll be honest, I considered going in there and adding some additional references to avoid the noise that comes from these kind of comments, but have decided not to.
You’re finding things based on your own beliefs, and that’s great, but everyone knows that the purpose of this post is far from any sexist / feminist reference. The sentence you reference has absolutely no relation at all to what you’re alluding.
Like you Jennette, I find it troublesome that a comment like this would even come up.
In case it wasn’t clear and avoid further confusion:
This is not about Women or Men. Blacks or whites. Muslims or Christians. Democrats or Republicans.
Several years ago on my own blog someone called me out for using language that was racially insensitive in one of my posts. At the time I rolled my eyes and was like, “Whatever,” and didn’t even bother replying to the comment. But looking back on it now I realize that person wasn’t attacking me or judging me, they just had a legitimate point that I could have used better language to express myself without unintentionally alienating part of my audience. My original comment on this post was left with similar intentions.
Hey @jennettefulda:disqus
I really don’t mind at all, it’s just one of those things that I am always amazed by. I suppose it’s just the open world we live in now. By amazed I mean how something like this even comes up at all in a post that is so far from it, but I guess like everything we all look and read things with our own life views and perspectives.
In my opinion, it’s impossible to write anything without potentially alienating some audience. I’m not saying I’m indifferent to it, I am saying that I expect it.
I assure you that your point is heard loud and clear though.
Thanks
Tony
Great write-up Tony.
Ah you made it back.. hope the beers treated you well.. :)
Amen! The other day I was reading an article about the Russian hacker group that recently stole 1.2 billion passwords (http://pxl.st/1q4epoR) and the author says, “My credit card information has been stolen four times in the last three years … I fully expect to have to get a new card at least four more times in the next three years. … Still, I feel comfortable buying things online … [b]ecause I have come to expect security incompetence from everyone and everything.”
While I am definitely a WordPress lover, I don’t stick my head in the sand and pretend that it is fool-proof. Denial and sugarcoating don’t do anyone any good. It’s common sense that the most-used CMS on the web is going to be the most targeted, and that vulnerabilities *will* be discovered and exploited.
In my opinion, the best course of action is to accept it, do everything we can to protect ourselves from it, and prepare ourselves for when it *does* inevitably happen.
Tony,
First, Thank you for serving our country.
Second, Thanks for the clear spoken, succinct presentation.
I have been attempting to be an “end user” for two years. Looked at WordPress as a business
website solution 2 years ago, decided no, it’s not as easy as “everyone” says, and passed.
I have again been evaluating/considering using WordPress and may jump in to install and do a test
(I envision locking myself up for a week). I can not begin to explain how confusing and “noisy” it is
from where I sit. And recent changes in various platforms – I’m looking at an e-Commerce business –
reading, exploring, frameworks, themes, plug-ins, pricing, support, updates, ratings, interactions,
“playing nice”……Installing, security, hosting, ……….It’s taken a lot to get to the point where I think
I now have a reference of understanding – and I’ve only scratched the surface. Oh, and yes, everyone is an
expert.
I have shared-hosting as I am an entry level one man band running on a bootstrap budget.
I am a business person first and foremost – who wants to get work done and run a business.
I do not have a budget that allows me to use hosted ecommerce, (I dislike the SAAS model
I prefer to “own” my business, site, tools); or to hire someone to do it all for me.
In another lifetime, I was a technical person (PM, VP, Sales, etc) in data communications corporate
networking & processing (think T-1, multiplexors, X.25) and used to be exceptional at wrapping my head around technologies and evaluating multiple solutions with different approaches. I have been in marketing and
outside sales forever, and previous business owning in commercial construction. Point being I’m grey matter
capable.
Now, I find trying to learn a new technology, platform, software, payment processing, api’s, browser, web hosting shix – even a new photobackup service – is a major endeavor. Endless posts/ articles with numerous links out to other articles, videos, and my personal unfavorite: zendesk and all it’s other iterations. NO manuals, documentation just endless posts on single topics…Ad nausium. No wonder I find it difficult to get back to the business of business and move forward- I’m being rabbit holed to death!
So, back to WordPress specifically. I look forward to further insights and strategies from yourself,
Sucuri, and a handful of others I’m still developing as straight shooting resources after plowing through
reams of paper…….Now if I can get up the courage to try it out………..
Thank You.
.
Hi Debbi
Thank you for sharing your thoughts. I’ve always been the type of guy to jump into the deep end of the pool, even before I knew how to swim. I’d say just give it a run, you might be surprised at how good you float.. :)
Tony
I’m happy for my quote to be attributed, you don’t have to hide it’s from me. I stand by those words on the importance of “updates” being the most important long-term precursor to real security. I think Microsoft and Google would agree with me if you look at how they approach updates now in their platforms like Windows and Chrome. (Apple has also gotten good at updates, which now happen in the background for apps by default, and of course every web service.)
Hi Matt
Thanks for your note. I didn’t attribute anyone’s quote as the purpose of the post was not any one individuals as much as it was a community as a whole. I’ll be happy to attribute though.
I also think that the point about Updates might be getting lost in the write up. The intention is not to say it is not important, on the contrary, it is very important, especially for the everyday end-user.
The challenge and purpose of the write up is more with the mindset and overall communication a whole community, which can be lost. When we begin to attribute success to a specific factor, especially in security, that’s where we get into trouble. The beauty of updates, is that they address knowns, it’s biggest weakness is it doesn’t address unknowns. That however is a much bigger conversation and so while Updates are important, they are but a small piece in a much bigger pie. I’ll do a follow up to provide more thoughtfulness on the subject.
I agree whole-heartedly on what you say about Microsoft and Apple, and that is truer to my point about the post as a whole. Take into consideration the two you mentioned, Microsoft and Apple, their approach is so much more involved – it’s not just an emphasis on update, it’s an emphasis on good security posture as a whole. That right there is and should be our message.
Thanks
Tony
Updates are the best way to address unknown unknowns, IMO. I agree it’s impossible to simplify to just one thing, but given that all code is fallible, updates are the one thing I would pick if I had to. Also when done right it’s one of the most user-friendly things out there, which in securing a system is always the toughest thing.
Fair opinion, but here is some food for thought..
In actuality addressing the unknowns challenge is achieved through posture. It’s through a concept of Defense in Depth that you achieve the most effective posture. It’s through things like updates, backups, auditing / monitoring, detection, protection and extra emphasis on access control… etc… that is how you address the unknown problem.
I worry that sometimes in our effort to oversimplify, providing that one simple solution, that is where we take the end-user down the wrong path. That is the trap I believe us to be in today..
Have you read any of the work of Nassim Taleb, like Black Swan or Anti-fragility? He writes more about finance and economics, but it’s really about people and systems, and it has influenced my thinking a lot on this topic.
I think Jennette makes a good point about alienating audiences, even if its unintentional.
Yes, so do I.
I don’t see where there has been a lack of disclosure or why it is wrong to emphasize updates over auditing. That seems absolutely the right emphasis for very practical reasons. Of course everyone should be doing both, but updates are the only simple option anyone and everyone can do or just have done automatically.
There’s an interview you did (Tony) with WPengine a few years ago where you said timely upgrades are the #1 priority in a list of the five most important security steps anyone can take. (http://wpengine.com/2013/04/24/how-tony-perez-of-sucuri-sets-up-his-own-security/) This was great advice and may be worth reprising now, but I note it does start with timely upgrades. My understanding of your criticism of MailPoet and WP core’s recent patches is that they didn’t emphasize their importance enough, but in both cases just getting the update out fast was the most important response. Most people won’t read about the upgrades no matter how they are worded.
I doubt anyone would disagree that because WP is easy to set up and speaks to a largely non-technical community of users on their terms, it is popular with hundreds of millions of people who are chronically under-educated about important technical issues impacting security and much else. But you also seem to go further and suggest that people who know better are not doing enough to educate end users and meet some undefined (Drupalish?) standard for a security disclosure. Can you elaborate on that? When WP sites are exploited on a massive scale, maybe some in the WP community circle the wagons and defend the platform, blame some third party code or the users, and a lot of people may lack the knowledge to really add much to the issue. That’s natural, but if it’s more superficial than helpful I’m not sure what you are offering that is substantially different. I also wouldn’t agree that security is an avoided or suppressed topic, but I agree with you that most popular security advice you can find for WP is superficial and not very security minded at all. There is very little that resembles your WPE interview, for example, which I have often recommended as educational reading.
Should there be a clear standard for how all security releases are classified, described, and communicated? Since WP is not the type of community where its public voice is typically developers speaking developer to developers what do you propose as the mass communication solution for end users?
Personally I think here is a lot more to the problem than simple ignorance that can be cured with communication and education about technical subjects. Lots of insecure sites are the result of people who simply don’t care and never will, sometimes because they just don’t have the resources or wherewithal. I’m not sure how that can be addressed with third-world users, but for everyone who can afford at least $150-350/year for hosting there is no reason anymore for them to be trying to manage their own server or use some inherently less secure type of shared hosting. I’ve seen some recent reports showing that hosts dropping the ball account for one of the highest categories of hacks on WP sites, and that certainly squares with my experience and awareness.
If there is any target to poke for lack of openness and accurate disclosure that impacts security on a massive scale, my vote would be for the cozy relationship that has always existed between WP and certain hosts with long, bad records, and sub-par ways of doing things that non-technical end users don’t know about. Educating users about quality in hosting (which has nothing necessarily to do with size, money and sponsorships) is dangerously close to criticism of people you’re in bed and need to some extent — and they aren’t likely take kindly to such “educational communication.”
HI Dan
Thank you for your very thoughtful response. To be clear, my intent was never to say that updates are not important, the point of the post was a different conversation all together and not really meant for end-users. This will likely be the first follow up post I share to better clarify the point – endusers, yes, update, update, update.. :)
As for my critique of the WP core, and MailPoet, it actually nothing to do with updates. I actually don’t think I mentioned in MailPoet in this post, did i? I thought my issue with core in their release was in how it was categorized, not how it was updated. That specific section, where I injected the reference, was to emphasize brand protection not so much is or isn’t important to the end-user.
My discussion in this post or elsewhere around updates has nothing to do wit it’s importance, it has to do with communication of what is important and how we should be educating the users.
As for this:
No I didn’t, you’re stretching here. My reference was to Drupal above was around categorization and how they handle, and nothing to do with end-users.
I’ll be writing a series of posts, hopefully, that addresses these points directly:
and
Like in anything, there will always be some segment of audience that will never care or will be indifferent to the entire process. That however should not dissuade us from trying and improving our communication.
By the way, this is a bad mindset:
A lot of the issues that were often attributed to shared hosts is actually no longer the case. The days of mass compromises because of shared environment is actually a few years behind us. It doesn’t mean that there are some bad hosts out there, but some of the biggest like Dreamhost, BlueHost, FatCow, HostGator, etc… do a very good job.
Great point at the end.
Take Care and thanks for stopping by.
I’m an end user and what some people refer to as a ‘mom blogger’ who jumped in the deep end almost 5 years ago and I’m still fighting off the sharks everyday. A self starter I was never afraid of WP. I studied and learned and eventually launched 2 more sites. Back then I was oblivious to the potential time suck (and stressor) that security threats bring. I didn’t consider the pitfalls like Debbi, nor did I stay educated about security and eventually my site was hacked last August. I scrapped my blog and started over with help from my webhost, who watched me flair for six long agonising months. I am a blogger and a photographer, I know nothing of the proper identification and removal of malicious script with no budget for security services. The silver lining? I learned how to not become ‘low lying fruit’, how crucial it is to update, remove unused themes, plugins, and change HTaccess configuration. I installed WordFence Pro, deployed 2 step sign in and reduced my plugin dependency. I chalked the whole thing up to WP experience and was fine with it until last week when my site was hacked again. I’m dumbfounded. Clearly the protection installed was not enough or I had plugin conflicts or maybe my theme had vulnerabilities. Who knows, at least this time I don’t feel so panicked, but I will say I am beginning to ask myself ‘is this worth it.’ Thank you Tony for the insight here and leaving the discussion open to everyone including end users.