Category Archives: Software development

Work around for Excel lose leading and trailing zeros when importing csv data

This is from reddit. “…open a blank workbook in excel, then from the Data tab, under the “Get External Data” setting, choose “from text” and point it to your file. This will open the text import wizard. On the first step, choose “delimeted” and then “Next.” On the second, select the checkbox next to “Comma” then click “Next.” Now in the third step, the field should be shown, along with Excel’s best guess as to the data type. You’ll see that your numbered fields are shown as “General.” Click the column containing the numbers and change that field to “text.” Do this for all applicable fields that you want to save leading zeros for, then click “Finish.” (Excel truncates leading zeros if it thinks a field is a number. Leaving it as text will preserve the zeros.)…

My workaround is to stick an apostrophe (tick mark) for the numbers I want to keep the zeros. In other words that essentially make it a text by adding the tick mark.

Sustainable software development II : rewrite and star performer

Developers usually like new project, sometimes called green field project, in the sense developers have more freedom to create (think of a white paper). In the same token, from time to time people will start a “full rewrite” project as the old project (code base) is “so messed up” and beyond repair. Is that real? A very high profile “rewrite” failure is the Netscape rewrite (see Joel Spolsky’s post here). And recently yours truly was involved in such “rewrite effort” as well. The result is also similar. I think not just developers, some managers also like the “rewrite” as well, due to various reasons. Some are valid, and some are not. Software (if it’s true software) usually lasts longer than people thought. In 2002, I visited my former workplace, a manufacturing place in Shanghai, and surprised to found they are still running my Foxbase/Foxpro based report generator, the the OS is Windows 3.1. At the time I worked on this (1994/1995), it saved 2 accounts, a few days of work (type calculators), each month. I bet they still used it for sometime after 2002, but I really doubt that will last till today, and the printer (Epison pin based) may not find punched hole printing papers :-)

Star Performer
I thought this problem for a while. I happened to listen to one podcast from “this agile life“, and read this article The surprising danger of being good at your job from yahoo/business insider, both helped me think more about this problem.

I think in software development world, it’s a very common phenomena, like @codinghorror said in this post “two types of progammers“, basically he is saying there 20% of developers are passionate about software development, 80% of them are there to get a pay check. But the star’s power and magic needs to be balanced or controlled, so that the teammates can also thrive and together the team can deliver something truly good.

Another related phenomena, is Lebron James in NBA. A few months ago we witnessed his heroics again. But Cavs did not win eventually. One reason I think is they are one man team. While their opponent Warriors are a real team.

Side note
Last week I received an email from my former boss asking me about a “magic number” I put in the code. I had to admit I messed up :-)

New job, offense and defense in software world

I changed job again recently, to be more precise, I made the change at beginning of the month. So today marks the conclusion of first 3 weeks.

I felt fairly good so far, both in terms of environment and the work. It’s a bit different from what I did in last few years, as I engaged more in product support (still software world). I consider this to be more defense type of work, esp. the part of trying to keep application up and running, vs. the development work I did in the past (more like offense). Both are important. Some new tasks include handling customer requests (technical, second or third level if we look it from level of support). We can say this is a bit like defense too, because sometime people literally threw problems or questions at you (or me).

I did give it some thought before making the change. I’ve been doing software development for the most part in last 15 years (first 8 years at Siemens PLM Software formerly UGS or Unigraphics). I’ve done some customer support here or there, mostly when I was with Autodesk. Most recently I was doing development at Mercy. I still like development. But at the same time, at this stage of my career, I like to think about what I am good at and what I like to do more. I think troubleshooting, and problem solving are definitely in my comfort zone. I also need to step out my comfort zone a bit (aka customer support). And I like to learn more on production, operation, devops (the new buzz words, essentially the automation of deployment and production), and last but not least security. So here comes my rationale. I think in some of the places like google, the job is really refined and one thing I am interested is “site reliability engineering“. Another good thing about current place is it’s a decent size company and has some interesting problems: both technical and non-technical. So here comes challenges.

Let’s see how I do as time goes :-)

PS: I heard about the news about my old company’s incoming layoff. We went through a similar exercise last year when I was there. At that time I was relatively new at the old place. And I was a bit scared. I think this expansion/shrinking is a way of IT life nowadays. Ironically, I got call from a recruiter and she told me the news I already heard. I recall last year some recruiter even sent email to my working account. Just a fact of post linkedIn world. Also another of my old company had a layoff recently too. See my post at uudaddy.

PS2: I think in software development there are also both defense and offense. Maintenance, fix bugs, and refactoring can be considered as defense. New features, bells and whistles can be considered as offense. Both are important as if we don’t refactor, or maintain, soon or later the code or the software will become unmaintainable :-)

PS3: in the soccer (or football outside of US), we have seen players moved from middle field to defender position as they age. Similar can be said for software work, from development to support, from new development to maintenance.

PS4: did Sony release Play Station 4, really? Just kidding. Another thing I want to say is this transition does not mean “I am no longer a coder”, a more precise description is I will do more diverse things, from coding to anything else related to product support (applications, and customer), whatever it takes. In a fancy term, I will get a more holistic view of application in production. :-)

Silly mistakes in web service implementation

Came across a strange problem this week using a web service from 3rd party. It’s a soap based web service. When we pass the XML with the required data elements in certain order, it works. If we did not follow that order, it complains invalid parameter or something. I guess in the code they have something like this:

if parameter_1 is invalid
return bad_parameter

if parameter_2 is invalid
return bad_parameter

In my opinion, we can do a simple fix as shown below:

if (parameter_1 is invalid) || (parameter_2 is invalid) || …

To be fair, I have made this kind of mistake myself, probably a while ago :-) I think it’s important for experienced developers have empathy towards people who make those mistakes, and understand how they come to the imperfect solution, and how to avoid those problems in the future.

(Update 05-09-2015) It looks like a common problem in web service, as explained in this thread. Quote:…Some vendors may grab SOAP parameter values by element name (the proper way) and others may grab the values by node position (what you’re probably running into)…

Another post on stackoverflow about similar problem on WCF SOAP web service.

Sustainable software development : I

I am starting a series of “Sustainable software development” blog posts. Like many series I did in the past, it could be a series of one and only, or it could be a true fall classic (WS series).

Back to the topic, I have been doing software development for almost 15 years, been through many organizations, sometimes move between different organizations inside one company. And I think among all these agile, water fall, off shore, near shore, on shore etc., the most important of all is: consistently deliver values to customer. The key of what I said is “consistently” and “value”. Note I am not trying to add another one or two buzz words here. By that I mean everybody should be happy: employee or contractor don’t get overworked, feel they got the short end of the stick, managers don’t feel they got squeezed between sales people and the end customer, the customer does not feel been lied to (by the “used car” salesman). How do we achieve that? I think it’s both a management and engineering question or discipline. Note I said “sustainable” in the topic, if any of the one conditions (symptoms) is true in real world, what will happen? Employee or contractor will leave, manager will feel depressed (they may not leave as fast because it’s usually harder to find a manager position), customer will get angry and make threatening calls…

Note I will mentioned “software”, not “application” or something. Software is a bit older word than application, or “apps” the new buzz word popularized by Apple (Steve Jobs?). Good old fashion software. A lot of times when I saw “applications” people developed, it’s merely a configuration or customization of some prepackaged software. I used to work for company does that sort of thing too. Don’t get me wrong, there is value in customization or configuration. But that is NOT “software development”, that is “software customization” or “configuration”. Also, there is a drawback to this approach, the “upgrade nightmare”, because it’s hard to upgrade to the original software due to poor customization. Customization, if properly done, I think the changes can be localized, and the application can be maintained. On the other hand, if it’s not properly done, just like “hard code” things in the code, it becomes harder and harder to catch up with the vendor as time goes. It’s very likely those kinds of system will generate a lot of outages and support calls, and we know most developers don’t like handle production support problem all day (and night). They like development better.

This reminds me a related question I came across on Quora How do you make programmers work 60-80 hours per week? The short answer is don’t do that. In today’s world, we still think about a problem when in shower (just like Warren Buffett got the idea to buy BoA stock in shower :-), as Ryan Matte said in Quora: a programmer never really turns it off. So from management point of view, they should really encourage the hard working developers to go off to take a break. I recall my middle school teacher had very similar philosophy.

So this is my 2 cents to get started on this topic. I will hopefully get back to this soon…maybe talk about how to use agile and team size to make software development more sustainable:-)

So long…

Java app memory leak, performance

A collection, I will add my comments later. Symptom: PermGen run out of space

Memory leak
Overview (Mark Thomas)

Memory leak happened in multiple deployment (more likely happens in development)

Tomcat 7 (Mark Thomas)

Application or library code that can trigger this situation include:
JDBC driver registration
Some logging frameworks
Storing objects in ThreadLocals and not removing them
Starting threads and not stopping them

There are also a number of places that using the standard Java API can trigger a similar issue. These include:
Using the javax.imageio API (the Google Web Toolkit can trigger this)
Using java.beans.Introspector.flushCaches() (Tomcat does this to prevent memory leaks caused by this caching)
Using XML parsing (the root cause is unknown due to a bug in the JRE)
Using RMI (somewhat ironically, causes a leak related to the garbage collector)
Reading resources from JAR files

Overview and use VisualVM to debug memory leak (Colm Divilly)

Nikita Salnikov-Tarnovski

Overview of memory and performance of J2EE app (PDF, Steven Haines)

Is pair programming worth the hassle?

I have been doing programming for over 14+ years, and from time to time I did pair programming (mostly debugging problem). Since early last year I have been doing more pairing, mostly due to the hints from above. I have mixed feeling on this, mostly along the lines of advantages of disadvantages talked about here by Mark Weldon.

Quote Mark here, “When you pair program, you’re effectively joined at the hip with your pair. You can’t pair if only one of you is there. This means that you both come into work at the same time, you both take lunch at the same time, you both take breaks at the same time, and you both leave at the same time. The work is so concentrated that you work 8-hour days (which is good). But you can’t take time off without affecting your pair. Working from home is possible, but it’s obviously not as pairy as sitting beside one another…” I have very similar feeling here. Mark is probably an experienced developer. I also did a google search, saw other people with different background shared their thoughts on pair programming (manningdigital, a CS student). Mark also talked about the personality clash, essentially one developer with strong will (or more stubborn gets her/his way), which is also echoed by other 2.

Curious about this topic, and don’t want to limit by the google, I came to quora, and searched this topic. I came up with Is pair programming worth the trade off in engineering resources and Does Pair Programming really work.

And I saw this from the legendary coder Kent Beck, he suggested it works when

“The problem space is big and unknown. You don’t know what exact problem you are solving and the probability is high that any solution you find will change your understanding of the problem.
The solution space is big and unknown. You are working with technology you don’t fully understand and the chances are that any solution you find will change your understanding of what constitutes an effective solution.
You need to build relationships on a team…”

I think that’s where I agree with on pair programming at this time.

Moving wordpress from shared host to aws ec2

I did an experiment recently trying to move my wordpress website from a shared hosting site to Amazon aws ec2 micro instance (t1). The migration was mostly successful, I did not make the move eventually due to some technical and non-technical reasons. Nonetheless, I would like to share some of the lesson I learned from this process. I followed this blog post form smashmagazine as a blueprint for “moving wordpress”. Note I already have ubuntu installed on ec2, and has setup keys for my laptop.

1) Setup wordpress on new host. This is the link I followed to setup wordpress on ubuntu on my aws ec2. Another useful article is here.

2) Upload files: I used FileZilla (SFTP) for uploading the files. Here is the link regarding setup keys. Note the file permission is also very important to make wordpress work, which is especially important when moving the sites. More on this later.

3) File permission, one thing I noticed quickly is I don’t have any writing privilege after moving, e.g., update plug-ins etc. Did google quite a bit, and tries to change the file permissions using FileZilla. No luck. Eventually I read this from wordpress official doc. Quote:…(a) file ownership: all of your WordPress files must be owned by the user under which your web server executes. In other words, the owner of your WordPress files must match the user under which your web server executes. The web server user (e.g., “apache”, “web”, “www”, “nobody”, etc.) is not necessarily the owner of your WordPress files…

I did “ps -ef|grep httpd” at my ubuntu server, and find the user for apache web server (not “root”). And changed the file owner to this user (via “chown” command).

4) MySQL server crash: I found this happens quite often (as often as once every 3 days). Turns out to be a memory problem for the t1 micro instance and apache web server and mysql fights for the memory, eventually mysql lost. The solution I found is add virtual memory to the system, and reduce the memory requirement for mysql in config file. Eventually I was able to have mysql run as long as 13 days without shutting down. Not ideal, but still meaningful improvement (13 days uptime vs 3). Another idea, people talked about is use a lightweight http server such as this one.

5) MySQL database migration. Found out for large database, php export will not complete, and will end up with some junk html in the database file (.sql), it will fail during import. The workaround is do the export and import in the mysql command line. But I still have one problem with database, it appears the Chinese character got lost during this process. This problem along with the mysql server problem mentioned above, and the relative higher cost of AWS compared to shared host, made me decide to not switch at this time.

Other links:
Setup php/mysql, and phpMyAdmin.

Craig Emerson has an up-to-date article on this topic WordPress CloudFront CDN Setup Using W3 Total Cache.

Update sqlite database during iOS app update

We all know updating sqlite database on an existing iOS app could be tricky, esp. with any schema (table) changes. In the good old days when I was developing in-house app, I found out the app would crash when I added a column to the sqlite database, after merely updating the app. The problem was, the old database won’t get updated if we don’t explicitly do it during app start-up (after app update). So what did I tell my user to do? Delete the app, and re-install the app. Not the most user-friendly way, and what if the user has some data on it that he/she wants to keep? What if this app have many users?

I had this updating issue in my mind for a while, because one of my apps needs to refresh its data. There are a few ways to do it: 1) Use a web service to get the data; 2) Just update the sqlite database. The former has some limitations: we could get data via web service, but note we still need to manipulate the database if we need to add a column or table (this is same for both approaches). The latter is faster in terms of development. So I decided to do it. I did some research on stackoverflow and found this post as my reference. Here is my implementation, if you are interested. The following is done in viewDidLoad.

Continue reading

Rockstar developer

I have thought about this topic for a long time. I remember seeing at one place that an exceptional developer can do work usually done by 10 developers. Or in other words, he/she is a rockstar developer.

Besides productivity, I think rockstar developer has the following traits:

1) Willing to share the knowledge with fellow developers, keep in mind we all learn from each other, rockstar can learn from (dare I say) ordinary developer too;

2) Attention to the details and code quality, and other good development practice such as TDD (test driven development), again sharing knowledge is applicable here;

Continue reading