Downloading source code from SVN/Git repository over HTTP

    Sites hosting open source projects provides an online viewer for browsing source code without actually checking out source code using SVN/Git clients. Checking out entire repository will take long time. Also with Git there is no straight forward way to checkout only a particular directory. Cloning Git repos takes long time as Git downloads entire repository to local machine. Even with sparse checkout, Git downloads entire repository. When bandwidth is a concern, one cannot checkout entire repository.

    One can use GNU Wget to recursively download files from online code repositories. For windows this can be downloaded from http://users.ugent.be/~bpuype/wget/

    Command to download a directory and its child directories and all files in it recursively excluding index.html is below. This will not download parent directories and files from external sites.

wget --cut-dirs=2 --level=15 --include-directories=src/main/java --recursive --no-parent --no-host-directories --reject=index.html -e robots=off --no-clobber  http://sourcesite.com/src/main/java

    Be careful with slashes. When I used backslash, it did not work.

--cut-dirs=N This ignores directories of N levels from root directory of the URL
--level=N Downloads files from N level of directories. Default is 5 levels
--include-directories=src/main/java Include only this directory and its child directories
--recursive Recursively download
--no-parent Do not go to parent directory of the given URL.
--no-host-directories Without this option wget creates a directory by the host name of the server
--reject=index.html Do not create index.html file
-e robots=off Exclude robots.txt when crawling the site
--no-clobber Do not overwrite existing files


    Here are few of the source code repository addresses.

http://svn.apache.org/repos/asf/
http://selenium.googlecode.com/git

    For Google Code sites use projectname.googlecode.com/git for Git repo or projectname.googlecode.com/svn if it is a svn repo.


Comments

Post a Comment

Popular posts from this blog

Multi Tenant applications using PostgreSQL Row Level Security

 PostgreSQL supports Row Level Security (RLS) so that applications can restrict who can see which records in a table either based in DB user name or based on Connection session variable. This feature can be used to implement discriminator based multi tenant where records for multiple customers are present in single table. This blog explains steps for implementation https://www.crunchydata.com/blog/row-level-security-for-tenants-in-postgres Advantage of this approach is that its easy to implement and day to day application development activities dont have to bother about breaking multi tenancy because code for applying discriminator is implemented at a framework that obtains connection from connection pool. Application developers dont have to modify individual SQL statement or even pass around tenant_id as a parameter in method calls. Good thing about RLS is that RLS policy can be applied to specific Roles in Postgres and these roles should be used by application connections wh...
Read more

Recovering Dell Laptop Windows OS using factory image from Linux bootable USB drive

I used Linux booted from a USB drive to restore Dell Windows 7 factory image on Inspiron 1545 when Windows 7 stopped booting and F8 key stopped working. This was done using Linux equivalent of Imagex command and used dell factory.wim file. Credits goes to - https://www.dell.com/community/Windows-General/Restore-from-dell-recovery-partition-with-Imagex-Solved/td-p/4083636 Software used 1, PuppyLinux BionicPup32 8.0 (Ubuntu Bionic 32 bit) Downloaded as ISO image from their website. It is good to use any other popular distros like Ubuntu. 2, When using PuppyLinux, download Devx package which has GCC compiler for the corresponding PuppyLinux distribution. ( http://wikka.puppylinux.com/devx ). When using a full fledged distribution, this step may not be necessary. 3, USB pen drive or any flash storage. I used SD memory card because my laptop will not boot from a pendrive. 3, unetbootin-windows-661.exe 4, Wimlib 1.13.1 - downloaded source because a binary distributio...
Read more

How to find actual Tomcat instance in a cluster

It is common that in a cluster we have to find out which Tomcat instance is used so that we can monitor the log files of that particular server only. When Tomcat is used as the application server, normally load balancing is done using Apache JK connector (mod_jk). JK connector adds name of actual instance to JSESSIONID cookie so that the connector can know how to route the request to the server which created the session. This works only when sticky session is used. When sticky session is not used, name of actual server is not appended to cookie and request is send to a server picked based on the load on each server. In this case, it is very difficult to identify actual Tomcat server. When sticky session is not used, the solution I can think of is to use a filter that will set a cookie with a value that identifies unique name of the Tomcat instance. This filter should have configuration option to disable enable this cookie based on a configuration. Any changes to the configurati...
Read more