Leave line endings alone

I have never had so much “fun” with line endings until today. What started out as a simple investigation in TeamCity eventually turned into an exploratory trip into the maze of git’s treatment of line endings. Here is the recount.

“Louie, we have a technical debt, there are a lot of ssis tests failing in TeamCity, can you take a look?”

“Sure”

The tests are all simple and similar in nature. They just call ssis packages to load test files into tables. Upon inspection I know immediately that this is an environmental issue as the tests reported are usually the ones we never have problems with on the local dev environment.

I open up the ssis project in Visual Studio and inspect the first package. The flat file manager is configured with {CR}{LF} as the row delimiter, check. I locate the test file and check its line endings in Notepad++, and that’s also fine, CRLF everywhere.

And indeed the test works on my local box.

I remote into the TeamCity agent and locate the same test file. Unsurprisingly LF snakes down the screen when I click on “Show All Characters”. Bingo.

I think to myself, all I need to do is to tell TeamCity to pull the files down and convert eol to CRLF, but how? There is a switch to “Convert line-endings to CRLF” but that only solves 99% of my problems as some files don’t ask me why in the test set have to have LF as line-endings. Back to square one.

.gitattributes comes to mind.

Going through blogs on the topics of autocrlf and gitattributes, my mind is filled with joy as I scroll down the posts written by wise men before me. I apply the settings religiously and again and again my hope turns into disappointment. The files in TeamCity still have lines ending in LF.

Out of desperation, I start to search for words that randomly pop up in my head. Few searches later, I am blessed with the revelation below:

TeamCity indeed doesn’t support .gitattributes.

Oh my god! Does that mean we have no other options but to accept failure. Of course not! My mind starts to mull and ponder. How about…

How about we use .gitattributes to control the eol behaviour for all dev environments and ask TeamCity not to do any eol conversion by switching off “Convert line-endings to CRLF”, good idea.

.gitattributes now looks like this:

# Don't do any eol conversion during checkin and checkout
 *.[Cc][Ss][Vv] -text
 *.[Tt][Xx][Tt] -text

TeamCity no longer converts line-endings to CRLF. All good but there is one thing left to do. I need to modify line endings for all test files once and for all.

I overwrite the files (with correct line endings, most with CRLF, some with LF), I add, I commit and I push. At last, I am blessed with a sea of green tests in TeamCity again.

–oo00oo–
Further readings:
https://help.github.com/articles/dealing-with-line-endings/
http://stackoverflow.com/questions/11116636/how-do-i-ignore-file-extensions-in-git-regardless-of-case
https://teamcity-support.jetbrains.com/hc/en-us/community/posts/206822315-Binary-files-being-corrupted-by-Auto-CRLF

Git aliases I can’t live without

This is a dump of my most used git aliases for easy future reference.

[alias]
 # log with pretty formatting
 l = log \"--pretty=format:%C(yellow bold)%h %C(red bold)%ad %C(white bold)%an %C(reset)%s%C(cyan bold)%d\" --all --decorate --date=short

 # Show commits from me (Louie) 
 la = log --author="louie" \"--pretty=format:%C(yellow bold)%h %C(red bold)%ad %C(white bold)%an %C(reset)%s%C(cyan bold)%d\" --all --decorate --date=short

 # Show history (commits) of a file
 h = log --follow \"--pretty=format:%C(yellow bold)%h %C(red bold)%ad %C(white bold)%an %C(reset)%s%C(cyan bold)%d\" --all --decorate --date=short

 # Show files in a commit
 f = show --name-status \"--format=%C(yellow bold)%h %C(red bold)%ad %C(white bold)%an %C(reset)%s%C(cyan bold)%d\" --date=short

 s = status
 a = add -A
 b = branch
 dif = difftool --cached --dir-diff
 cm = checkout master
 cd = checkout develop

How to create a new local git branch that tracks a remote branch

I’ve had to look up this on StackOverflow twice in the last week. I thought it’d be a good idea to jot down some notes so I can remember what to do next time.

1. Add the remote repo
git remote add louie git@github.com:louiebao/dbarsenal.git

2. Get all the branches for the new remote repo
git fetch louie

3. Create a new branch locally that tracks the remote branch
git checkout -b demo louie/demo

4. Check
git branch -v -v

How to setup GIT for Windows using Command Prompt and Posh-Git

On my local environment, I am using:

  • Git for Windows
  • Command Prompt (comes with Windows so no additional install required)
  • Posh-Git (for nice visual prompt)

I have had the above setup for quite some time now. After helping out a colleague with the setup today I have decided to put the instructions in a post so I can quickly reference it later.

Install Git

Everything starts from here. You can download Git for Windows from here if you haven’t already got it installed. Make sure you ticked “Use Git from the Windows Command Prompt”.

git_setup

Setup Command Prompt shortcut

The steps below will provide you with a MUCH more friendly command prompt.

1. Pin Command Prompt to the taskbar.

2. Right click on Command Prompt (the one in the taskbar) and click on Properties. Set the properties to:

Target: %windir%\system32\cmd.exe /K “cd C:\Dev & powershell”
/K = Carries out the command specified by stirng but remains
& = Used to separate multiple commands

This command will 1) Switch into C:\Dev (or any other working folders) and 2) Run powershell.

3. Click on the Options tab. Tick QuikEdit mode in Options. This makes copy and paste operations a lot easier.

Bonus Tip: In order to start cmd.exe in admin mode, you can add the following registry key:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers]
"C:\\Windows\\System32\\cmd.exe"="~ RUNASADMIN"

Install PowerGUI (optional)

You can choose any IDE to work with powershell scripts but this is my personal preference. PowerGUI is a free IDE for powershell scripts that can be downloaded from here.

Install Posh-Git

1. Download the Posh-Git zip file from here. You can install Posh-Git using other methods like git clone or PsGet but my experience says manipulation of a zip file is the easiest way for beginners to understand and follow.

2. Unzip (using 7-Zip) to C:\Dev\posh-git-master. If you don’t have 7-Zip and relied on the Windows “Extract All…”, you might encounter issues with powershell’s execution policy. Instead of messing around with the execution policy, just go download and install 7-Zip from here. It will save you lots of headaches.

It doesn’t matter where you extract the files, I leave it in C:\Dev because that’s where I keep things.

3. In a command prompt, go into the posh-git-master folder and run .\install.ps1. You may or may not need to set the powershell’s execution policy but RemoteSigned at the LocalMachine level should be sufficient.

4. Inspect your newly created profile. If you don’t know where your profile is located. Just run $profile in the command prompt. You profile should contain two lines that read:

# Load posh-git example profile
. 'C:\Dev\posh-git-master\profile.example.ps1'

5. By now you should get the nice prompt that’s integrated with git. command_prompt_posh-git

6. Bonus step. I have talked about including the current date time in the command prompt in a previous post. After installing posh-git, the handy timestamp in the comand prompt would be lost. In order to get that back, I had to edit the file C:\Dev\posh-git-master\profile.example.ps1 by adding Write-Host($(Get-Date)) above the default Write-Host line:

...
Write-Host("")
Write-Host($(Get-Date))
Write-Host($pwd.ProviderPath) -nonewline
...

Setup SSH Key

1. Generate a new SSH key. Again, seemingly simple step but expect unpleasant surprises if your environment is not configured properly. Make sure:

  • C:\Program Files (x86)\Git\bin is in your PATH.
  • HOME = C:\Users\<user name> is added to the system environment variables.
  • C:\Users\<user name>\.ssh folder is manually created if it doesn’t exist already.

Once all the above checks passed, run the command below in the command prompt:

C:\Users\<user name>\.ssh> ssh-keygen -t rsa -C "<email address>"

Tip: If ssh-keygen.exe is not in the path already, you’ll need to look for it in one of the bin sub folders under the git folder.

2. Press <Enter> to accept the default location and file name.

3. Press <Enter> and <Enter> again when prompted to enter a passphrase.

4. Open the public key file (C:\Users\<user name>\.ssh\id_rsa.pub) and copy the content.

5. Add the public key to GitHub / BitBucket.

–oo00oo–
For more info, please refer to the links below:
Better Git with PowerShell