Posts Tagged ‘msysgit’

Docker for Windows (beta) and msysgit

Friday, April 15th, 2016

I’ve recently joined the beta program for Docker on Windows (now based on Hyper-V).

I wanted to keep my current config using msysGit but got weird errors when executing Docker commands from msysGit: https://forums.docker.com/t/weird-error-under-git-bash-msys-solved/9210

I could fix the issue by installing a newer version of msysGit with support for the MSYS_NO_PATHCONV environment variable. With that installed, I then changed my docker alias to a better approach:

docker()
{
    export MSYS_NO_PATHCONV=1
    ("$DOCKER_HOME/docker.exe" "$@")
    export MSYS_NO_PATHCONV=0
}

Hope this helps!


My development environment on Windows

Thursday, July 30th, 2015

TL;DR Use Bash on Windows like me and you’ll be in heaven too, with penguins and ice creams :)

Update 2015-08-07: I now use ConEmu rather than Console2; apart from this, my configuration is still as described below


In this post I’ll describe my current Windows configuration and development environment. I’ve already covered how I’ve customized my Windows 10 install, but here I’m going to explain what I do to have an efficient workflow and a ‘portable’ configuration.

For those wondering, yes I use Windows as my main operating system (don’t throw the tomatoes just yet). As I’ve said in the past, I do prefer Linux, but I also enjoy gaming and dual boot is just not for me anymore. Moreover having one OS across all the machines I use at home and at work (apart from tablets) is useful.

Before going in the nitty gritty details, here’s a brief overview of my setup:

  • CloudStation (Synology NAS application): synchronizes files between my desktop, NAS, laptop & tablets; you can substitute this with Dropbox or whatever else you like
  • Git (i.e., msysGit) & git bash: because I love git and MSYS
  • Console: a great Windows console enhancement (supports multiple tabs, different shells, different fonts, easy text selection, shortcuts, …
  • AutoHotkey: create macros & scripts. I use it to show/hide my bash console with the ‘²’ key
  • bash profile: if you know *nix, you know this but I’ll cover the basics below
  • a ton of portable apps (or non-portable ones adapted)
  • GitHub: great Git client for Windows

Here are a few examples of things I can do with my setup (on all my Windows machines):

  • hit ‘²’ and start typings commands
  • use common *nix commands such as ls, cat, less, sed, …
  • type ‘e’ and have the File Explorer opened in the current folder
  • type ‘s’ and have Sublime Text opened
  • type ‘s cool’ and have Sublime Text opened with the file ‘cool’ opened it in
  • type ‘npp’ and have Notepad++ opened
  • type ‘n’ and have my notes opened
  • type ‘m’ and have my GMail mailbox opened
  • type ‘g cool’ and have a new browser tab open with the Google search results for ‘cool’
  • type ‘imdb shawshank redemption’ and see the IMDb info about the best movie of all times
  • type ‘wiki’ and have my wiki opened
  • type ‘f’ and get facebook opened
  • type ‘nlfr echt waar’ and have google translate opened with the translation of ‘echt waar’ from dutch to french
  • same with frnl fren …
  • type ‘img mario’ and see pictures of Mario all over
  • type ‘mkcd test’ and have the test folder created and cd into it
  • type ‘ws’ and have WebStorm started
  • type ‘idea’ and have IntelliJ started
  • type ‘write 001’ and have my 001 project opened in Scrivener
  • type ..3 and be 3 levels higher in the file system tree
  • type ‘p’ and have my bash profile opened for edition in Sublime Text
  • type ‘mindmap’ and have my Mindmap opened in FreeMind
  • well you get the idea … :)

The goal of my setup is to strictly limit the number of steps to get my development environment up and running (e.g., after I get a new device or need to reinstall one) and to synchronize my configuration(s) between all devices I work on.

At the heart of my configuration, there is CloudStation, the synchronization app provided by Synology NASes (best NAS devices you can find on the market). I use CloudStation to synchronize the following between my devices:

  • Configuration files (I’ll cover these later)
  • Programs
  • Books I’m currently reading (or plan to read soon)
  • Comic books (only thing that I synchronize w/ my tablet)
  • Pictures (e.g., wallpapers & pictures of my face — if I need to upload one somewhere)
  • Podcasts
  • Book drafts (stuff I’m writing from time to time)
  • My notes.txt file

The most important parts are the config files and programs because that’s the core of my setup.

My CloudStation folder is organized as follows:

  • _FOR_HOME: stuff to bring back home
  • _FOR_WORK: stuff to bring to work
  • _NOW: stuff I’m currently busy with
  • Books
    • Reading
    • Later
  • Configuration
    • Bash: contains my bash profile
    • Dev: contains the configuration for all my dev tools
      • Eclipse: my Eclipse preference files (.epf), code style rules, etc
      • IntelliJ: my portable IntelliJ config (config & plugins)
      • WebStorm: my portable WebStorm config (config & plugins)
      • Git
      • Templates: project templates
    • Home: my *NIX home folder (.gitconfig, .npmrc, .ssh folder, etc are in there)
    • Scrivener
    • XBMC: my portable XBMC config (worth another post in itself)
  • Electronics: my current electronics projects
  • Guitar: that thing with strings that I learn when I find free time (i.e., not often enough)
  • Lightroom: my LR catalog (worth another post in itself)
  • Music: things that I listen again and again
  • Pictures
  • Podcasts
  • Programs
    • dev: JDK, maven, docker-machine, groovy, intellij, mongodb, nodejs, npm, python, eclipse, svn, webstorm, …
    • electronics: arduino IDE, atanua, circuit, fritzing, …
    • emulation: zsnes, project64, …
    • games: minecraft and other small games ;-)
    • readers: e-book readers & comic book readers (e.g., ComicRack)
    • seb: my own tools
    • tools: a huge ton of (portable) apps
    • writing: apps like Scrivener, WriteMonkey, …

To give you an idea of the tools I have in CloudStation, here’s a part of what I use:

  • SublimeText: my current preferred text editor (no it’s not VI, I’m more of a nano guy)
  • Notepad++: my previous preferred text editor
  • SysInternals suite: greatest Windows toolkit ever
  • 7-zip: it does it all
  • KeePass: one passphrase to rule them all
  • kitty: putty portable replacement
  • ADExplorer
  • AntMovieCatalog (again worth another post)
  • AutoHotKey (more on it below)
  • borderless window tool: useful for games that don’t have a fullscreen windowed mode
  • calibre: manage my e-books
  • Console2: awesome Windows console replacement (more on it below)
  • desktops: obsolete with Windows 10 :)
  • ffmpeg: holy grail (or so I thought)
  • exiftool: dump exif
  • ext2explore: let me see EXT partitions
  • fat32format: format stuff
  • folder2iso: sudo make me an iso
  • freemind: can’t live without mindmaps
  • guiformat
  • HexChat: coz IRC is still there in 2015 (yeah I’m tired looking for the links ^^)
  • hfsexploer
  • hxDen
  • ImageMagick: do me some magic with images
  • JDownloader: download all the things
  • jude: sometimes helpful for quick UML drawings
  • libmp3lame: encoding stuffz
  • mplayer: who can live decently without mplayer around?
  • MySQL Workbench
  • netcat: netcat for Windows, weee
  • PortQry: check UDP ports
  • Privoxy: local proxy
  • ProxyGet: dumps info about the currently configured proxy (useful in locked-down environments)
  • PS3Splitter: split large files
  • restoration: restore deleted files (family helper)
  • SolEol: download subtitles easily
  • SQLite
  • SteamMover: move steam folders around
  • SubtitleEdit: fix me thy subtitles
  • twt: CLI for Twitter
  • USBDiskEjector
  • uTorrent
  • wakeMeOnLan: wake up LAN devices
  • wget
  • win32diskimager: create img files
  • WinDirStat: where’s my free space gone??!
  • WinMerge: diff me up
  • WinSplitRevolution: can’t live without this to re-arrange/resize windows around
  • winscp
  • YNAB: yes you do need a budget

Okay so all of that currently sums up to about 30GB so clearly the initial sync time is quite long because there are a huge amount of very small files to sync, but once synchronized, you get an idea of all I have available at my fingertips.

There are apps that I do actually install on my OS for two reasons: either the app is way too large to be copied around or it integrates too deeply with the operating system. Here are some applications that aren’t in my CloudStation folder:

  • Lightroom
  • Photoshop
  • VLC: much easier to install & have the file type associations
  • Winamp: same idea
  • Git
  • Spotify
  • Google Chrome
  • Steam
  • Battle.net
  • Daemon Tools
  • CrashPlan client
  • Dropbox
  • GitHub client
  • VirtualBox

Okay, so far you have an idea of the stuff I carry around with my but you don’t know yet how I use it all. Let’s assume for a moment that my PC goes up in flames and that I need to setup a brand new one.

Here’s what I need to do in order to get back up and running (to the point of being able to work):

  • install the OS (haha)
  • install drivers (hoho)
  • install CloudStation and let it sync all my files
  • install msysGit
  • create a .profile (bash profile) file in my home folder with the following contents
    • source /c/CloudStation/Configuration/Bash/bashProfile.txt
    • this loads my actual bash profile which is part of my CloudStation synchronized files
  • add AutoHotkey to the startup list (i.e., put a shortcut under ‘shell:Startup’)
    • C:\CloudStation\Programs\tools\AutoHotkey111502_x64\AutoHotkey.exe
  • copy my AutoHotkey script to the Documents folder
  • at this point I can already hit ‘²’ and my console opens up, with all my tools available
  • install the few other apps I like to have
  • done!

Okay so let’s see how my console is set up.

So when the OS boots, it now starts AutoHotkey. I’ve written a small script that opens up Console2 when I hit ‘²’ and hides it when I hit ‘²’ again, a bit like Quake’s console or Yakuake under Linux (although I don’t have the nice animation ;-)).

Here’s the script

; QuakeConsole, used in combination with Console2
; Change your hotkey here
;SC029 == ²
SC029::

DetectHiddenWindows, on
IfWinExist ahk_class Console_2_Main
{
 IfWinActive ahk_class Console_2_Main
 {
 WinHide ahk_class Console_2_Main
 WinActivate ahk_class Shell_TrayWnd
 }
 else
 {
 ; put the console at top left
 WinMove, 0, 0
 ; show the console
 WinShow ahk_class Console_2_Main
 WinActivate ahk_class Console_2_Main
 }
}
else
 Run "C:/CloudStation/Programs/tools/Console-2.00b148-Beta_64bit/Console2/Console.exe"
DetectHiddenWindows, off
return

The script is quite simple: if Console2 is running, pressing the configured key will either show/hide the window (and put it on the top left of the screen on show); if not, it starts the program.

As you can see Console2 is also in my CloudStation folder, just like AutoHotkey is.

So now, here are the relevant parts of my Console2 configuration (console.xml file placed in the Console2 program folder in my CloudStation tools)


	
		
			...
		
	
	
		
			
		
		
		
		s caption="1" resizable="1" taskbar_button="1" border="1" inside_border="5" tray_icon="1">
			
		
		
		
	
	
		
		
		
	
	
		...
		 
		
		
	
	
		
			 
  			 
  			 
			
			
			
			
			
		
	
	
		   
		
			
			
			
				
					
				
			
		
	

With this Console2 configuration:

  • I automatically get in a Git Bash shell in my workspace (i.e., where I have my project folders)
  • I can hit ctrl+ F1 to open a new tab, also with the same Git Bash shell
  • I can use ctrl + c / ctrl + v to copy/paste
  • I can use the middle mouse click to paste
  • I can use shift + click to select & copy text

Okay thus so far you can see how I can use my bash shell on Windows, I just hit ‘²’ and I can enter commands, not much harder than hidding the Windows key to open up the start menu & searching for stuff, except that in my shell I’ve got all my commands & aliases available.

The final (and most important) part of my configuration is my bash profile. It’s where I configure my environment, define functions and aliases, configure programs in my path, etc. 

Given how long it is I’ve create a Gist of it here (note that it’s just a subset of my whole config): https://gist.github.com/dsebastien/47d24a5d6c1b8005f434

I don’t claim any rights on the functions I use in my bash profile as it is mostly based on stuff I gathered over time from various sources. Though there mustn’t be many crazies like me to do this kind of things on Windows ;-)

In the file, you’ll see that the basic principles are quite simple & straightforward, folders added to the path, functions and aliases with shorter names, etc.

I have tons of ideas to improve my bash profile but I just don’t have time now. There are just tons of worth-looking examples all over the place if you’re interested. And if you have ideas/links, just post them in the comments!

What I like with my current configuration is that it is portable in the sense that if I add a new tool or change the configuration of an existing one, all my machines are directly updated.

Of course it is far from perfect, most apps aren’t up to date and the more I add into my system, the harder it gets to update stuff. Package managers are indeed the solution and *NIX has had them since the dawn of ages, but there’s hope on Windows too.

In the future, Windows package managers like OneGet (now part of W10) and Chocolatey should simplify things, but I don’t feel like it’s usable for my goals right now (correct me if I’m wrong).

The most evident and easiest solution would simply be to use Linux, but for as long as I’ll be playing games from time to time, I won’t go back. 

 


A bit of Windows Docker bash-fu

Monday, April 20th, 2015

In my last post I’ve mentionned that Microsoft has helped Docker deliver a native Docker client for Windows (yay!).

I’ve also promised to share the little bits that I’ve added to my Windows bash profile to make my life easier. As I’ve said, I’m a huge fan of MSYS and msysGit and I use my Git Bash shell all day long, so here comes a bit of Windows bash-fu.

For those wondering, I prefer Linux and I would use it as my main OS (did so in the past) if I didn’t also like gaming. I can’t stand fiddling around config files to get my games running (hey Wine) and I can’t stand losing n FPS just to stay on the free side. Finally I am not too fond of putting multiple OSes on my main machine just for the sake of being able to play. The least painful solution for me is simply to use Windows and remain almost sane by using Bash.

One thing to note is that my bash profile as well as all the tools that I use are synchronized between my computers in order to allow me to have a consistent environment; I’m done raging because I’m in the train and some tool I’ve installed on my desktop isn’t available on my laptop.. I’ll describe that setup.. another day :)

So first things first, I’ve installed Docker v1.6.0 on my machine without adding it to the path or creating any shortcuts (since I’m not going to use that install at all); you can get it from https://github.com/boot2docker/windows-installer/releases/latest.

Once installed, I’ve copied the docker client (docker.exe) to the folder I use to store my shared tools (in this case c:\CloudStation\programs\dev\docker). I have the Docker machine in the same folder (downloaded from here).

append_to_path(){ # dumb append to path
    PATH=$1":"$PATH
}
...
# Docker
export DOCKER_HOME=$DEV_SOFT_HOME/Docker
append_to_path $DOCKER_HOME

alias docker='docker.exe'

alias docker-machine='docker-machine.exe'
alias dockermachine='docker-machine'
alias dm='docker-machine'

export DOCKER_LOCAL_VM_NAME='docker-local'

In the snippet above I simply ensure that the docker client is on my path and that I can invoke it simply using ‘docker’. Same for docker-machine, along with a nice shortcut ‘dm’.

Note that I also set a name for the local Docker VM that I want to manage; you’ll see below why that’s useful.

docker-config-client()
{
	echo "Configuring the Docker client to point towards the local Docker VM ($DOCKER_LOCAL_VM_NAME)..."
	eval "$(docker-machine env $DOCKER_LOCAL_VM_NAME)"
	if [ $? -eq 0 ]; then
		echo "Docker client configured successfully!"
	else
		echo "Failed to configure the Docker client!"
		return;
	fi
}
alias dockerconfig='docker-config-client'
alias configdocker='docker-config-client'

The ‘docker-config-client’ function allows me to easily configure my Docker client to point towards my local Docker VM. I’ve added some aliases because I’ve got a pretty bad memory :)

This function assumes that the local Docker VM already exists and is up an running. This is not always the case, hence the additional functions below.

docker-check-local-vm() # check docker-machine status and clean up if necessary
{
	echo "Verifying the status of the local Docker VM ($DOCKER_LOCAL_VM_NAME)"
	dmCheckResult=$(docker-machine ls)
	#echo $dmCheckResult
	if [[ $dmCheckResult == *"error getting state for host $DOCKER_LOCAL_VM_NAME: machine does not exist"* ]]
	then
		echo "The local Docker VM ($DOCKER_LOCAL_VM_NAME) is known by docker-machine but does not exist anymore."
		echo "Cleaning docker-machine."
		dmCleanupResult=$(docker-machine rm $DOCKER_LOCAL_VM_NAME)
		
		if [[ $dmCleanupResult == *"successfully removed"* ]]
		then
			echo "docker-machine cleanup successful! Run 'docker-init' to create the local Docker VM."
		fi
		return
	fi
	echo "No problem with the local Docker VM ($DOCKER_LOCAL_VM_NAME) and docker-machine. If the machine does not exist yet you can create it using 'docker-init'"
}
alias dockercheck='docker-check-local-vm'
alias checkdocker='docker-check-local-vm'

The ‘docker-check-local-vm’ simply lists the docker engines known by docker-machine in order to see if there’s a problem with the local Docker VM. Such a problem can occur when docker-machine knows about a given Docker engine and you delete it (e.g., if you remove the Virtualbox VM then invoke ‘docker-machine ls’, then you’ll get the error).

docker-start()
{
	echo "Trying to start the local Docker VM ($DOCKER_LOCAL_VM_NAME)"
	dmStartResult=$(docker-machine start $DOCKER_LOCAL_VM_NAME)
	#echo $dmStartResult
	if [[ $dmStartResult == *"machine does not exist"* ]]
	then
		echo "The local Docker VM ($DOCKER_LOCAL_VM_NAME) does not seem to exist."
		docker-check-local-vm
		return
	fi
	
	if [[ $dmStartResult == *"VM not in restartable state"* ]]
	then
		echo "The local Docker VM ($DOCKER_LOCAL_VM_NAME) is probably already running."
		docker-config-client
		return
	fi
	
	if [[ $dmStartResult == *"Waiting for VM to start..."* ]]
	then
		echo "The local Docker VM ($DOCKER_LOCAL_VM_NAME) was successfully started!"
		docker-config-client
		return
	fi
	
	if [[ $dmStartResult == *"Host does not exist"* ]]
	then
		echo "The local Docker VM ($DOCKER_LOCAL_VM_NAME) does not exist. Run 'docker-init' first!"
		return
	fi
}
alias dockerstart='docker-start'
alias startdocker='docker-start'

The ‘docker-start’ function above tries to start my local Docker VM. It first assumes that the machine does exist (because I’m an optimist after all).

Since the docker-machine executable doesn’t return useful values, I have to resort to string matching; I know that this sucks but don’t forget we’re on Windows.. There’s probably a way to handle this better, but it’s enough for me now.

If the VM does not exist, the docker-machine check function is called.

If the VM cannot be started, it might be that the machine is already running; in that case the docker client gets configured (same if the start succeeds).

If the VM clearly doesn’t exist then the function stops there and points towards ‘docker-init’ explained afterwards.

docker-stop()
{
	echo "Trying to stop the local Docker VM ($DOCKER_LOCAL_VM_NAME)"
	dmStopResult=$(docker-machine stop $DOCKER_LOCAL_VM_NAME)
	#echo $dmStopResult
	if [[ $dmStopResult == *"Host does not exist"* ]]
	then
		echo "The local Docker VM ($DOCKER_LOCAL_VM_NAME) does not seem to exist."
		docker-check-local-vm
		return
	fi
	
	if [[ $dmStopResult == *"exit status 1"* ]]
	then
		echo "The local Docker VM ($DOCKER_LOCAL_VM_NAME) is already stopped (or doesn't exist anymore)."
		docker-check-local-vm
		return
	fi
	
	echo "The local Docker VM ($DOCKER_LOCAL_VM_NAME) was stopped successfully."
}
alias dockerstop='docker-stop'
alias stopdocker='docker-stop'

The ‘docker-stop’ function stops the local Docker VM if it’s running (pretty obvious eh ^^). In case of error, the docker-machine check function is called (docker-check-local-vm).

docker-init()
{
	echo "Trying to create a local Docker VM called $DOCKER_LOCAL_VM_NAME"
	dmCreateResult=$(docker-machine create --driver virtualbox $DOCKER_LOCAL_VM_NAME)
	#echo $dmCreateResult
	
	if [[ $dmCreateResult == *"has been created and is now the active machine."* ]]
	then
		echo "Local Docker VM ($DOCKER_LOCAL_VM_NAME) created successfully!"
		docker-config-client
		return
	fi
	
	if [[ $dmCreateResult == *"already exists"* ]]
	then
		echo "The local Docker VM ($DOCKER_LOCAL_VM_NAME) already exists!"
		dockerstart
		return
	fi
}
alias dockerinit='docker-init'
alias initdocker='docker-init'

This last function, ‘docker-init’ helps me provision my local Docker VM and configure my Docker client to point towards it.

With these few commands, I’m able to quickly configure/start/use a local Docker VM in a way that works nicely on all my machines (remember that I share my bash profile & tools across all my computers).

Voilà! :)


Docker, Docker Machine, Windows and msysGit happy together

Sunday, April 19th, 2015

Hey there!

tl;dr The Docker client for Windows is here and now it’s the real deal. Thanks MSFT! :)

If you’re one of the poor souls that have to suffer with the Windows terminal (willingly or not) on a daily basis but always dream about Bash, then you’re probably a fan of MSYS just like me.

If you’re a developer too, then you’re probably a fan of msysGit.. just like me :)

Finally, if you follow the IT world trends then you must have heard of Docker already.. unless you’re living in some sort of cave (without Internet access). If you enjoy playing with the bleeding edge… just like me, then chances are that you’ve already given it a try.

If you’ve done so before this month and survived the experience, then kudos because the least I can say is that the Windows “integration” wasn’t all that great.

Since Docker leverages Linux kernel features so heavily, it should not come as a surprise that support on Windows requires a virtual machine to host the Docker engine. The only natural choice for that VM was of course Oracle’s Virtualbox given that Hyper-V is only available in Windows Server or Windows 8+ Pro/Enterprise.

Boot2Docker was nice, especially the VM, but the Boot2Docker client made me feel in jail (no pun intended). Who wants to start a specific shell just to be able to play with Docker? I understand where that came from, but my first reflex was to try and integrate Docker in my usual msysGit bash shell.

To do so, I had to jump through a few hoops and even though I did succeed, a hugely annoying limitation remained: I wasn’t able to easily run the docker command using files elsewhere than under /Users/…

At the time, the docker client was actually never executed on Windows, it was executed within the VM, through SSH (which is what the Boot2Docker client did). Because of that, docker could only interact with files reachable from the VM (i.e., made available via mount). All the mounting/sharing/SSH (and keys!) required quite a few workarounds.

At the end of the day it was still fun to workaround the quirks because I had to play with Virtualbox’s CLI (e.g., to configure port redirections), learn a bit more about Docker’s API, …

Well, fast forward April 16th and there it comes, Microsoft has helped port the docker client to Windows.

With this release and combined with Docker machine which is also available for Windows, there will be a lot less suffering involved in used Docker on Windows :)

In the next post I’ll go through some of the functions/aliases I’ve added to my bash profile to make my life easier.