Base-Art

Phil's blog

en
Apr 2014
Mo Tu We Th Fr Sa Su
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30

Bringing RSS to SVN

About one week ago, Aaron Brady posted a svnstatus.py script which displays an SVN repository version history in an HTML document. I liked it and wanted to use it as a post-commit hook script for the XMLObject SVN repository. But svnstatus is not optimal at all. If you want to periodically build a static HTML file it's ok. The problem, if you want to use it in post-commit hook is that it rebuilds the file for each commit, which turns to be really CPU hungry.

That's why i wrote SubveRSSed.py. SubveRSSed incrementally builds an RSS feed containing the commit history of a SVN Repository. Then an HTML document is built based upon the RSS feed. This approach has many advantages:

  • post-commit hook friendly :)
  • the RSS can be made available for aggregators and people can monitor the activity/vitality of a repository
  • the history can still be consulted via the HTML document, if you don't like aggregators nor RSS ;)

But there is one issue thought, the RSS accumulates all revisions, so it can grow enormously. I think i'll make 2 feeds, one complete and one with the 20 latest revisions .. So SubverRSSed will evolve, may be in the PythonFR SVN repository :-)

BTW, many thanks to Aaron for svnstatus from which SubveRSSed used some functions and embedded CSS.

update: the script now handles 2 RSS feeds as described above :)

update2 : check out SubveRSSed at this url

update3(2005/05/05) : There's a more recent post about this script.

by Philippe Normand on Fri Sep 17 09:05:01 2004 (Viewed: 62828 / 43 comments )
  |   RSS  |   RSS2  |   Atom  |   Source  |   Edit

#.   phil on Sat Sep 18 00:04:20 2004

Now under Revision control in PythonFR !!

#.   Jkx on Sat Sep 18 18:16:13 2004

I think you should fix the url to point to the svn version ? This url is fact .

#.   Aaron Brady on Tue Sep 21 10:24:42 2004

And I was just getting around to doing some caching myself.

I guess now I don't need to - hooray lazyweb!

#.   OlDer on Tue Sep 21 15:04:52 2004
There is bug in the program:
header % TITLE substitution runs before setting TITLE value from commandline parameters.
So header always has default TITLE value - 'Subversion repository'
#.   phil on Tue Sep 21 16:02:39 2004
Thanks for bug report. I commited a new revision of SubveRSSed which (i hope :) fixes the problem.
#.   OlDer on Wed Sep 22 22:40:47 2004
Another bug. If there are records without author (if anonimous writes are allowed, for example) - it fails on parsing xml file in second and subsequent runs.
So should it be:

author = TextNode(optional=True)

in Item class?

And have you considered using pysvn instead of calling svnlook via popen?
#.   phil on Thu Sep 23 10:16:00 2004

OIDer,

You're right about the optional flag on the TextNode. I'll have a look to pysvn even if it makes another dependency for SubveRSSed.

Anyway thanks for your feedback :-)

#.   OlDer on Thu Oct 21 12:28:20 2004

I'm still here ;)

Found another issue. If subverssed.py can't execute svnlook program - it does not show error message, but shows usage help instead.

#.   phil on Thu Oct 21 17:02:17 2004

You're quite a good bugHunter .. I may have others softs for you :)

Corrected the problem in revision 11 of SubveRSSed.

#.   OlDer on Sun Oct 24 19:09:33 2004

Thanks. Is there some bugzilla around? ;)

So, I have another suggestion (not a bug). I'm running Apache and SVN on Windows. And while SubveRSSed get its data from the svnlook, and it is windows console application - it outputs all data in current Windows OEM codepage. And it would be nice if it insert appropriate http-equiv meta tag into the generated html. Or, at least, convert all html to current Windows ANSI codepage.

It is just random thought and I do not know correct way to implement it. And I know that python's locale support on windows is far from perfect...

#.   phil on Mon Oct 25 09:17:09 2004

About the bugzilla, there will be one on PythonFR probably ... one day :)

About your feature request i'm not sure to understand what tags do you want ? BTW i'm not using windows .. so tests could be hard to do under that OS ;) May be should we continue discussions by email ? i'm phil _at_ respyre [dot] org

#.   Jason Huggins on Fri Oct 29 23:29:37 2004
I get the following error when I try to run this file on Windows. (Windows 2000, Subversion 1.1.1, Python 2.3.4, PyXML 0.8.3, XMLObject 0.1.3, subverssed.py revision 13 from your repos)

Any ideas? Other than "Don't use Windows" :-)

C:\scripts>python subverssed.py -r C:\subversion_root\sandbox
Adding version ... 1
Traceback (most recent call last):
  File "subverssed.py", line 201, in ?
    main()
  File "subverssed.py", line 171, in main
    rss.channel.items.insert(0,item)
  File "c:\python23\Lib\site-packages\XMLObject\TypedList.py", line 98, in insert
    item = self.checkItem(item)
  File "c:\python23\Lib\site-packages\XMLObject\TypedList.py", line 75, in check Item
    self._compareTypes, it):
  File "c:\python23\Lib\site-packages\XMLObject\TypedList.py", line 39, in walkOnXMLObject
    theClass = classregistry.registry(registry).getClass(xoName)
  File "c:\python23\Lib\site-packages\XMLObject\classregistry.py", line 75, in getClass
    return self.classes[className]
KeyError: 'item'
#.   Jason Huggins on Fri Oct 29 23:38:08 2004
Hmmm... the problem went away when I downgraded to XMLObject 0.1.2... Go figure. Hope this is usefult information. :-)
#.   phil on Sat Oct 30 00:34:58 2004

Hum yeah XMLObject changed quite a bit between 0.1.2 and 0.1.3. And it seems that i forgot to adapt SubveRSSed for the last XMLO version. I'll fix that issue. Thanks for feedback.

Anyway Blake Winton showed me few days ago a nice SVN option for the "log" command. Try "svn log -v --xml" .. really fun :)

#.   phil on Sat Oct 30 18:22:36 2004

Re Jason,

I corrected the issue you had. It was only a typo mistake. SubvRSSed is now compliant with latest XMLObject package (0.1.3).

#.   Jason Huggins on Tue Nov 9 00:04:23 2004
Phil,

Yup, your fixes worked... all is well with  subverssed.py and XMLObject 0.1.3. Cool stuff.
#.   OlDer on Mon Dec 13 16:23:39 2004

There is another bug here. Just add file with spaces in the name and you'll see.

As quick'n'dirty solution I've changed lines 169-171 as this:

if len(line) > 1:
    k = line[0]
    v = " ".join(line[1:])

but this give wrong result when file name contains multiple consequent spaces ;-)

#.   phil on Mon Dec 13 23:11:49 2004

Revision 23 fixes the problem ... i don't like placing spaces on filenames, that was a furtive bug to my eyes :)

Though i don't understand why this fix won't work when file name contains multiple consequent spaces ?

Anyway thx for bug report once again :)

#.   OlDer on Wed Dec 15 11:33:52 2004

I'm also prefer to replace spaces with underlines, but there are others ;)

It will work with multiple spaces, but it will convert these multiple spaces to one space. For example "this       name" will be translated into "this name".

#.   luke on Thu Feb 3 01:57:10 2005

Any chance I can get a quick guide on installing this script?? I am new to subversion and am setting up my own server and am not sure exactly on how to get this thing running.

Thanks.

#.   phil on Fri Feb 4 10:26:41 2005

Luke,

About hooks, it's all explained in the SVN Book. About SubveRSSed, it's quite simple. Create a post-commit hook with the following line on it:

$REPOS_NAME="fooBar"
/usr/local/bin/subverssed.py -t "$REPOS_NAME" -r "$REPOS" -o "/var/www/" -v "$REV"

Thus, the RSS feeds will be incrementally updated (-v option). If you want to create the feeds from a big (many revisions) repository, do the same on the command-line but without the -v option, and replace $REPOS with the full path of your repository on the filesystem.

#.   Luke on Fri Feb 25 03:08:53 2005

Thanks for the help, but I am still having problems. I get the following from the command line...

/usr/local/bin/subverssed.py -t "$REPOS_NAME" -r "$REPOS" -o "/usr/local/apache/htdocs/rss/" subversion/libsvn_fs/fs-loader.c:136: (apr_err=160033) svn: Unknown FS type 'bdb' Problem executing 'svnlook' ... Exiting.

Previous to that I have set both REPOS_NAME and REPOS to their correct values. Any ideas?

#.   cw on Thu Mar 3 20:11:00 2005
Hello there,

Few questions? I know everything except python. Some quick questions if you a a hot min. As of now I have svn installed working to the T and everything else working crazy silly, Bug tracking, CCNET, and this is one of my last challenges. I would like to install the python script you wrote on my server to have it read the commits to the repos and output them in xml or rss feed. I saw the posts and have started to build a test on my laptop to make sure I know what I am doing. Drop me a line and let me know what programs I need and where to place the .py file you wrote. Of course I have to edit the code to fit my repos file names and website. Thanks for your time. I am going to stay at it and mess with it. I keep getting this stupis error when I run it:
 
>>> rss_svn.py
Traceback (most recent call last):
File "", line 1, in ?
NameError: name 'rss_svn' is not defined

I know this is stupid error because I have not done something right. Talk to you soon

C
#.   phil on Fri Mar 4 10:53:17 2005

Luke,

I think your subversion setup is a little broken .. The problem is that svnlook (a subversion tool) gives an error. I advise you to get help with the subversion people, give them the error you get. I can't really do more to help you :/

#.   phil on Fri Mar 4 10:57:19 2005

cw,

You need:

  • Python
  • EaseXML (easexml.base-art.net)

To install SubveRSSed, i advise you to look on the SVN book (cf link above). Search for "post-commit hook". To run a script do the following:

python /path/to/the/script.py
#.   cw on Fri Mar 4 18:08:34 2005
Thanks for the reply. What folder in python24 do I place these files in after I unzip them.
CW
#.   cw on Fri Mar 4 18:44:52 2005
OK, I have messed with this long enough for today. The question or questions I really have are where do I place the EASEXML folder at so the subRSSed.py script will see it when I run it. I keep getting the error cant find EASEXML. I have everyth ing else setup in svn. That has been setup like silly. Thanks again for your time. Python24 is going and just need to know where E
Cfolder is placed so the .py script will import it.
#.   phil on Fri Mar 4 20:02:18 2005

CW,

Search for a site-packages directory on your windows thing. Place the EaseXML directory on it and SubveRSSed should find it. Either way add something like that in SubveRSSed, before importing EaseXML:

import sys
sys.path.append('/path/to/EaseXML')
#.   cw on Sun Mar 6 17:22:16 2005
So, thanks again for the help. Now that you are so helpful I will ask you many questions if that is ok. One more I hope and the last one. When I put the path in for svnlook.exe on a windows based machine, I get an error:
C:\Python24>python c:\python24\rss_svn.py
Problem executing 'C:\Program Files\Subversioin\svnlook.exe'... Exiting.

I know this is a problem with some kind of system variable and I am not sure how to fix it and dont really want to mess with it to much because you are such a help. I have been messin with other crazy silly stuff and kind of burnt right now. Thanks again for your time.

CW
#.   phil on Sun Mar 6 17:34:07 2005

I'm not sure that popen is that portable. BTW you made a little typo mistake:

C:\Python24>python c:\python24\rss_svn.py
Problem executing 'C:\Program Files\Subversioin\svnlook.exe' ... Exiting.

Try: C:\Program Files\Subversion\svnlook.exe instead

#.   cw on Sun Mar 6 21:03:47 2005
Sorry about the same post over and over, the refresh for odd reason made it do that. Well I got it to work with just moving svnlook.exe to the c:\ now I get a write error on the output. I am still working on this. The output is suppose to go where and how is the file written. Thanks again. Also can you delete the 4 entries I accidently placed in there.
CW
#.   cw on Thu Mar 10 00:01:00 2005
Well I gots it working with one exception. When you run it again without deleteing the .xml or .html files it gives me a run time error. Is there a reason why or do you have to deleted the .xml and .html files before you run the python script again. This will be a pain in the ass, if this has to be done I got some ideas. Thanks again.

CW
#.   cw on Thu Mar 10 00:34:37 2005

I am not allowed to post anymore?

#.   phil on Thu Mar 10 09:57:37 2005

CW,

Can you contact me by mail ? phil at respyre dot org And give me the exception you have It will be easier to resolve your problem. You can still post on the "official" blog but not in the "dev" blog because the article is too old.

#.   cw on Thu Mar 10 21:17:10 2005

Can I have your email in normal terms. I am not sure what it is? I have tried all combinations. Thanks.

#.   on Wed Mar 30 14:46:20 2005

Phil: you need to specify the SVNLOOK variable with both ' and " quotes, like this: SVNLOOK = '"C:Program FilesSubversionbinsvnlook.exe"' Worked for me (W2k, ActiveState python).

#.   phil on Wed Mar 30 17:05:12 2005

You mean triple quotes ? BTW where are (anti) slashes in your SVNLOOK ?

#.   Mahesh on Mon Apr 25 14:44:23 2005
hi phil..
i've installed EaseXML0.2.0 in my system and copied your subverssed.py 1.0. I tried runing the script with the following line

> python subverssed.py -t "Test" -r file:///tmp/temp/trunk
Traceback (most recent call last): File "subverssed.py", line 106, in ?
class Seq(XMLObject): File "subverssed.py", line 108, in Seq
items = ListNode('ItemId',name="rdf:seq")
TypeError: __init__() got an unexpected keyword argument 'name'"
----------------------------------------------

I got this error may be because of mismatch between the EaseXML versions. is that so? which version of EaseXML I suppose to install?
#.   phil on Mon Apr 25 15:26:00 2005

Mahesh,

SubveRSSed 1.0 requires EaseXML SVN revision 105 or higher. I will add a version checking in SubveRSSed to prevent this "issue".

For now, check out latest EaseXML copy at:

http://svn.base-art.net/public/easexml/trunk/

#.   Ronnie on Mon Jun 13 12:16:15 2005

Something to share with those who is trying to run the script in Windows, upon installed the newest set of EaseXML and the script, and when I execute "c:python24python.exe subverssed.py", the following error message shows up,

Problem executing 'C:Program FilesSubversioinsvnlook.exe' ... Exiting.

And I found that by chaning the value of "SVNLOOK" to c:\progra~1subversion\bin\svnlook.exe

The above mentioned issue goes away.

One side note, if single slash is used, the error message shown will become "c:program filessubversioinsvnlook.exe" (note the typo), no idea on why though.

Cheers, ronnie

#.   Glenn on Wed Sep 14 21:41:12 2005

The above format didn't work for me. svnlook, svnlook with multiple quotes, or the full path to svnlook with safe filename. What did work for me was to use the full name

svnlook.exe

#.   Nik on Sat Nov 26 17:30:31 2005
Hey. Everything seems to work fine but eventually i get this error. Im using the lastest version from svn:

  File "./subverssed.py", line 342, in main
    changed = helper.get_changed(i,encoding)
  File "./subverssed.py", line 271, in get_changed
    for path, action, base_rev in self.get_changed_files(rev):
  File "./subverssed.py", line 263, in get_changed_files
    changes.append([path, action, change.base_rev])
UnboundLocalError: local variable 'action' referenced before assignment

#.   phil on Thu Dec 1 10:08:59 2005

Hey Nick,

Sorry for late reply i was away during 4 days. I fixed the error you got, very odd though.

Comments not allowed anymore on this post