Category Archives: Doing It Wrong

Congratulations American Airlines

Posted by mysticstache on

I did not think it possible. American Airlines has beat all the odds and actually made the center seat more hellish! Some of you may not remember, roughly 10 years ago there were several law suits, some not valid, brought against airlines for the cramped travel conditions. Doctors even had a name for the injury done to frequent fliers who had to sit in the wrong position for so long. American Airlines, almost immediately, yanked out 2-3 rows of seats to spread them all out giving everyone more leg room.

On Monday I flew 4 hours in the center seat of a 1 month old aircraft ran by American and I can safely say the  leg room has been reduced back to pre “we want your business” levels. Not only was the distance between the seats ridiculously small, but the space under the seat in front was reduced 1/3 to accommodate some idiotic metal box that was bolted to the seat support that was also offset 3+ inches from where it should have been.

There was no lumbar support in the seat back. I was writhing to gain relief between lower back pain, upper back pain and calf spasms the entire journey. They only make the aisle seats available to customers with status. Well there’s a great way to discourage a person from flying with you enough to earn status. They’ll be wheelchair bound by 25k miles.

PostgreSQL, AWS, and Musical Bottlenecks

Posted by mysticstache on

I have had the misfortune of working with PostgreSQL for the last 8 months. Working is a relative term, for me little work has been done mostly I’ve been kicking off queries waiting forever fo the returns and then trying to run down the bottleneck.

I am not a Linux professional and have to rely on those professionals to diagnose what’s going on with the AWS instance that runs PostgreSQL 9.3. Everyone who looked at the situation has had a different opinion. One person looked at one set of performance data and said the system isn’t being utilized at all, someone else would say it’s IO bound, still someone else would say it’s the network card… So we wnet through all these suppositions added more RAM, then more processors, then we used the SSD drives more, finally switching from Non-provisioned IOPS to Provisioned IOPS got the system roughly as far as we could push it to where the complex queries would drive one CPU Core to 100%.

Now those of you who work with read enterprise RDBMS might say, “Wait… One CPU core reached 100%?” Well yes, of course, because you see PostgreSQL does not have parallel processing. Yeah…

No matter how many CTEs or sub queries present in a query statement sent to PostgreSQL, The processing of said query will happen in a synchronous, single threaded fashion on CPU core. I’m thinking SQL Server had parallel processing in the late 90’s or early 2000’s? It’s 2014 for crying out loud.

And it gets better! According to my observations, the Postgres process is also single threaded. This process is responsible for writing to the transaction logs. So there isn’t any benefit to create multiple log files for software striping and efficient log writing. In fact, one big insert seemed to back up all the smaller transactions, while the first insert wrote to the transaction log.

This is one of the joys of Open Source offerings. If the development community doesn’t think a feature is important you have to fork the code and write the feature yourself. What blows me away is that companies are willing to gamble the success of their products and implementations on something so hokey.

Open Suck… I mean Open Source

Posted by mysticstache on

If you’re reading this for a socialist country, I’m sorry but you’re going to struggle to understand the basic premise of this discussion. The application of a common cliché in capitalist societies, “You get what you pay for” I believe is universally appropriate. From my father-in-law, who bought the cheapest satellite service and complains incessantly about how much he wishes he had the same cable service I have but is unwilling to pay the higher service charges, to out sourcing call centers to regions of the world that speak a different language than the users of this service, to booking a cheaper hotel near the Orlando amusements with free shuttle service that’s just a glorified, overcrowded city bus without the graffiti. Going cheap is almost always going to disappoint. But this is a technical blog and my focus is Business Intelligence.

I’m working on a favor for a friend and I wanted to take this opportunity to explore some new technology. This friend of mine doesn’t have any budget for this project so I’m looking for cost effective components for this application that’s simply client front end to an RDBMS. My friend runs a small collection of Windows 7 desktops, I love Entity Framework, I’m proficient in Visual Studio, and I don’t need a “Big Data” solution. So I start thinking Open Source. Alright, hurdle 1, I’m not a java guy, and some of you might start harping about how Ruby, Rails, PHP running on Apache, Beans and Java all vastly different things…. I’m not into any of them; they’re all Java to me. A lifetime ago I played with swing and it sucked on Windows. Most Java apps I see run in Windows, are crap.

I don’t want to go into an in depth discussion on all the options, but I decided to investigate PostreSQL based on a recommendation from someone in my network who swears by it. One of the things I liked is the multi-OS support. Just in case the world turns upside down and I want to install the database one something other than a Microsoft OS, I thought I’d work with an RDBMS that would work the same no matter where it was installed with ne common client. The installation was smooth enough. I installed everything and clicked next, next, next… no errors. Good. Then I started researching ADO .NET clients to support Entity Framework, that’s where the wheels fell off.

In the realm of free providers to go with the free RDBMS; there is an OLEDB provider pgnpoledb, multiple JDBC drivers, and one ODBC/.NET provider npgsql. Now, I’m skeptical man and before I went down the path of actually trying to connect Entity Framework to the PostgreSQL database I decided to read the npgsql wiki. Pages were devoted to all the different issues and bugs, what was or wasn’t being submitted for acceptance in GitHub. From the headache mounting on my cranium, I could tell this option was going to require maybe a bit more effort than I was willing to invest in a favor for a friend. A lot of posters were pointing to the .NET provider for PostgreSQL from DevArt. Long story short, $199 for what I wanted… Wait a second I thought this crap was all Open Source and free!

Let’s just explore this concept, which has long been my complaint with the Open Source stack. If your goal is to create a mission critical high availability enterprise application with the Open Source offerings, you must be prepared to not only code your application, but also the platform on which it runs, or abandon the “Potentially Free” benefits of Open Source by purchasing licensed products to augment and stabilize the Open Source platforms. Option 1 means roughly doubling your workforce or your time to market. You need resources to code the platform and resources to code the application or resources that do both, but really only one at a time. Option 2 cuts into your equipment and tools budget and you need to verify what the vendor’s royalty and redistribution requirements are. No one wants to depend on a component that requires $1000 royalty for every user on a 40,000 seat client server application, right?

There are other Open Source challenges I love to joke with the diehard apologists I know. Like the fact that your favorite platform was written by one talented foreigner who doesn’t speak your language and only responds to email questions once a week when the internet service satellite flies over his bunker. I like a challenge as much as the next person, and I sympathize with the desire to revolt against the powerful software companies that are so slow to accommodate user needs. But, I’m just not willing to chance providing a service, where contractually I have to pay a refund for every minute of down time, dependent on a platform that was developed by hobbyists and amateurs.

Look at the example I stated above where the free provider has lots of challenges and the paid one is stable and supports all features of the toolset it’s meant to service. Developers whose livelihood (paycheck) is dependent on the successful execution of a project are naturally going to be more motivated to generate a better product than those who are working merely to support a community. Likewise, those tasks that facilitate the collection of said paycheck will take priority over the needs of a community, which leads you to have more down time as you wait for someone to get off from work (or high school marching band practice and homework) to fix a bug in the platform your product depends on and publish it to GitHub.

 

 

Job Req. Sanity Check

Posted by mysticstache on

Let me start by saying I am not an HR guy. Nor have I ever been a full-time recruiter of any sort. So perhaps, I’m way off base with my thoughts on this topic. PLEASE straighten me out if I am because there are a lot of practices within this space that make no sense to me.

I.  The Skill Set Years Experience Mismatch

Lately I have seen a flood of open position postings on the various job boards that will say something to the effect of “Jr. Developer\Recent College Grad\1-2 Years experience” as the headline of the posting. Only to find in the requirements section, experience (which to me means more than just exposure or reading a help doc online) for some 30 different technologies. Maybe, yes maybe with the right set of circumstances a Jr. resource as described in the headline might have started in an environment where he or she was given free rein to provide solutions through whatever means. I was lucky enough to have started my career as the only software developer for a successful Insurance company where I was able to explore whatever new technology came along and experiment with different techniques. I think this is pretty rare. Some companies spend the first 6 months breathing over a new resources shoulder with weekly code reviews before they’re promoted to level on and the code reviews come when the developer is ready. Many companies only let their resources sustain existing code and teach them just the basics to troubleshoot the existing technologies while the more senior staff works on innovation.

So are the hiring managers or recruiters looking for 80% of the required skills? One or two? Software design and development professionals are detail oriented and precise personalities. If I can’t talk about every skill listed, I’m not going to apply for a position.

II.   Competing Technologies

Another favorite of mine is when the laundry list of experience includes market competitors. The posting is looking for someone with 5 years experience and expert knowledge of Oracle, DB2 and SQL Server, or Expert level .NET and Java. First, can you really become expert in 5 years, especially if the maybe 2 of those you were just doing maintenance work (i.e. spell checking websites)? Secondly how many companies invest tens of thousands of dollars in SQL Server and more tens of thousands on Oracle? As a vendor software developer your product may need to support more than one database platform. However, what percentage of the candidates the job market hail from vendor software companies? Are there really any transferrable skills between .NET and Java? It seems to me trying to grow one resource into an expert of both is far more expense than cultivating two specialists and most companies would do the latter.

These types or requirements lead to a lot of confusion for candidates. They don’t know if they should bother applying or not. The recruiters are inundated with resumes that don’t fit the request from the hiring customer.

III.   Automated Recruitment Phone Recruiting

This year in particular I have been flooded with outsourced call center recruiter calls. These calls always follow the same format.

  • I answer the phone to silence
  • A few seconds later someone in a very thick accent says, “Hello may I speak to George?”
  • “Yes this is George.”
  • Faster than any normal human being should be able to speak -“Uh hi. My name is gibberish. gibberish gibberish gibberish gibberish gibberish gibberish gibberish gibberish gibberish gibberish gibberish gibberish …”
  • Me, “Whatever you’re talking about I’m not interested. Thanks.”
  • Hang up.

It’s as bad as the campaign calls around supper time during an election cycle. Who in their right mind thinks this is in any way an effective means to find a qualified candidate? I seriously doubt these individuals understand the technical requirements well enough to successfully phone screen much less are able to fight through the language barrier well enough to have a real conversation about the candidate or the opportunity.

IV.   Don’t Read the Resume

Another new interesting fishing tactic is the mail blast, or I guess that’s what’s going on. Why else am I getting emails for Jr. or Intermediate 5 years or less positions from the job boards where my resume clearly showing 16 years of experience are posted? Or the expert Java Architect roles I was sent when Java J2EE doesn’t appear anywhere on my resume? Recruiters, does this tactic work?

I understand there is a perception in the US job market right now that a lot of people are out of work and some companies are hoping to cash in on getting better qualified candidates for less compensation. This perception has created a recruiter feeding frenzy atmosphere. The truth is most of the top ranked talent is aware of what’s going on and they’re sitting this cycle out, or contracting. The unemployment rate among software development professionals is not nearly as high as other skill sets like manufacturing and construction. I believe this tactics will not be successful, and my land your corporation with a lot of negative feedback on a site like GlassDoor.com.

First, Know What You Don’t Know

Posted by mysticstache on

This post goes under “Doing it Wrong”, and it describes a situation where a data model… got out of hand.

A colleague of mine recently said, “When developing a new software product the first 6 months of code are stellar. The team is really focused on getting everything as right as possible. After that first six months the pressure is on to produce something and start making money. Overnight the focus changes to the delivery date and every corner is cut.” No truer words were spoken of the project that is the subject of this article. In the beginning there was a dedicated team for the design and development of the data model to be implemented in SQL Server cobbled together from the more senior developers. This team had complete control over the data tier. They performed their duties with diligence, creating volumes of documentation and naming standards.

One small challenge, however, was that this team didn’t actually have the skills required for such an undertaking. This deficiency manifested itself in the reliance on a renowned modeling tool that was originally designed for Oracle. This tool included SQL server support in later revisions, but it was SQL Server support written by Oracle experts. One of the “features” of this product was automated generation of indexes for parents and children of Foreign Key relationships. The output of this feature created non-clustered indexes for clustered indexed fields. There are several Knowledgebase articles on MSDN explaining why this is a worst practice, but I’ll summarize by saying, “unresolvable deadlocks under load”. Oracle doesn’t have clustered indexes; in fact there are several differences between Oracle and SQL Server that drive professionals to specialize in one product or the other and many skills do not translate between the two platforms.

Then came crunch time; having been perceived as the “bottle-neck”, the data tier team was broken up. Each member joined a different development team where it was assumed they would perform the same function as before, but concentrated on their team’s realm of responsibility. Now there was no more coordination of the data model design and each team was given full reign to create whatever entities they thought they needed. The naming standards were abandoned, the index issues propagated through “cut and paste”, data duplication ran amuck, and you name it.

Well, actually none of this was any issue really, until the company wanted to go after larger clientele. This monstrosity of a data model worked perfectly for small clients. In fact, no one took any issue with running some of the most poorly written custom reports ever conceived against the transaction database during business hours, until that large client signed the bottom line.

Suddenly, the application was not performing well. The product could not service the demands of a large customer. What should this company do?

They still had the chance to do what they should have done in the beginning. Involve some real expertise, either by hiring more knowledgeable technical resources, or working with a consulting team to get the product started in the right direction.

After three years of unstructured, unguided software development, the issues aren’t going to be resolved by the team basically capable of layering on more weak bandages and spaghetti code. My colleague again, “It doesn’t matter how much you tune it or modify it; you’re not going to get a sub-compact to perform like a Formula 1. If you have, you’ve replaced so much of the original that you should have just bought the better car to start with.”

Many enterprises aren’t concerned with the long term effects of making short term staffing decisions. Focus is usually placed on large head count over high talent. Highly talented development professionals not only have a higher level of quality output, but quantity as well. Such resources will already know the answers or the correct solutions to challenges unseasoned employees will generally research and then make the wrong choices on.

Minimally, recruiting a devoted product architecture team will set new efforts ahead in the game just by relieving the burdens of keeping up with new technologies from the development staff at large. A qualified SQL Server professional would have spotted the index issue immediately. That same professional would have been able to design a data model that would allow for the growth of features without the expediential growth of data model entities in such a way that product delivery timelines aren’t jeopardized.

Notice also I’m speaking of a SQL Server professional. If even within the realms of database experts individuals tend to specialize a particular platform. Is it not reasonable that someone who excels at writing compiled code is not going to naturally have the same success in developing data models? It has been my experience that even the best web developer will struggle to keep up with a mobile applications developer developing mobile applications. Additionally, neither resource is highly skilled at writing the best formed Transact SQL queries. Trying to form a data architecture team out of developers who haven’t specialized in data architecture is a losing proposition.

In summary, pick the right tool for the job from the start; be that a modeling tool focused on the platforms you want to use or a professional specializing in those same technologies. If the staffing cost seems too much at first, you might have a great opportunity for a temporary jump start form a team of talented consultants. In all honesty it will be cheaper than trying to turn a sub-compact into a Formula 1 race car.