Some words on things we like

mist.io in private beta

We’re excited to announce that mist.io is now in private beta! If you’ve already signed up, expect to receive an invitation in the coming days.

In the meantime, check out the screencast we've prepared. Big thanks to Diane Bisgeier from Mozilla WebFWD!

Plumi is critical for the commons!

Unweb.me has been working with the University of Southern California to rebuild www.criticalcommons.org, an educational video archive for fair & critical participation in media culture.

We are excited to write about www.criticalcommons.org going live after smooth cooperation between Anna Helme, Steve Anderson, Erik Loyer and Unweb.me. The new version adds significant improvements to the overall functionality and stability of the site, plus allows easy uploading of media files and commentaries.

criticalcommons.org main page

Critical commons is a video site built on Plumi. It is a collaborative online teaching tool for cinema studies, and a project designed to push the boundaries of fair use in an educational context.

Critical Commons was initiated by the University of Southern California School of Cinematic Arts Institute for Multimedia Literacy, and was officially launched in 2009 at the Digital Media and Learning and Open Video Alliance conferences. It was originally built by EngageMedia and Infinite Recursion on an earlier version of Plumi.

Functionality

On Critical Commons an educator can upload clips, attach commentaries and build lectures based on these clips. Other users can add commentaries to existing clips throughout the library. Clips can be embedded in other websites and, at the same time, it serves as a media repository for Scalar, an open-source media-rich scholarly publishing platform.

criticalcommons.org browse media page

criticalcommons.org media view page

Features

New features include the following:

  • easier registration for advanced users (users fill in the form and admins get informed to approve the application),
  • easier addition of media files (video, image, audio), commentaries and lectures,
  • addition of genres and browsing through genres, for media files,
  • improved search functionality,
  • all features of Plone 4.x, that make it faster, more stable and lightweight, with improved UX etc,
  • the features of Plumi 4.x, for example the mediaelementjs HTML 5 video player with Flash fallback and the new transcoding daemon of Plumi.

     

    criticalcommons.org search page

    criticalcommons.org upload media page

    Migration approach

    The previous Critical Commons website was built on Plone 2.5 and a heavily customized fork of Plumi 0.2. We reimplemented all the customizations on top of the latest versions of Plone and Plumi in the form of two seperate open source components for Plumi 4.x, one for the content (criticalcommons.content) and the other for the skin (criticalcommons.skin)

    Our goal has been to keep existing functionality, improve user experience, make one site viewable on desktops, laptops and mobile devices and migrate all previously published content. We exported all content from the old site in JSON format and then imported it on the new site.

    Code

    The code has been released in the following two packages:

    Critical Commons Skin: https://github.com/plumi/criticalcommons.skin

    Critical Commons Content: https://github.com/plumi/criticalcommons.content

    Plumi also benefited from this. We made improvements to the transcode daemon, tested Plumi on mobile devices and committed code that will be contained in the upcoming Plumi 4.5 release.

    Further plans

    Future plans for development include the addition of user channels, better integration with social media and use of Popcorn.js to enable time-based interactive media.

    We would like to thank Steve Anderson and Critical Commons for giving us the chance to participate on such an interesting and inspiring project.

    Introducing plown: Plone security tool

    Plown is a security tool for Plone CMS.

    Plone is one of the most powerful and secure CMS that has been around for many years. Very few vulnerabilities exist for Plone, comparing to other major CMS, as we can see on the stats taken by the National Vulnerability Database and assembled on http://plone.org/products/plone/security/overview

    But despite the fact that there aren't the hundreds of sql injections and XSS attacks that are disclosed every now and then for the big PHP based CMS, even the most secure system can be penetrated due to misconfigurations, use of weak passwords and if the admins never apply the patches released.


    Plown has been developed during penetration tests on Plone sites and was used to ease the discovery of usernames and passwords, plus expose known Plone vulnerabilities that might exist on a system.

     

    What Plown does

     

    • Username enumeration
    • Multithreading password cracking.You can specify the login url (if different that login_form) and the number of threads (16 default)
    • Known vulnerability enumeration, based on urls/objects exposed. If found vulnerable, the tool informs about the vulnerability and the url of the patch
    • Version enumeration is planned, based on md5 hashes of static content (css, js)

     

    Plown is written on Python and the code is available on github

    $ git clone https://github.com/unweb/plown

     

     

    Nagios clients behind paranoid firewalls

    Acting proactively is one of the key rules to save you from an infrastructure malfunction. However, hosting providers these days are paranoid about firewalls -- unfortunately, monitoring ports do not belong in the list of standard open ports. Whether you have a paranoid provider, a private data center not visible from the outside world, or a home server behind NAT your servers can be monitored using Nagios. In the next few lines we discuss our solution to this problem!

    Nagios infrastructure monitoring

    Rather than waiting for someone calling you because their server is down due to a disk crash, a broken upgrade or a weird behaviour of the system, having email alerts for such cases can save the day. As numerous organizations do, we use the excellent Nagios free software to monitor servers and get alerts for any system or service misbehaviour. Nagios operates on a classic server-clients model, where clients run a service that listens for connections from the server on port 5666.

    We wanted to add another Nagios client, but port 5666 is firewalled on this specific server and there was no way we could overcome the ISP's policy. Another option we considered is using another open port on that server, and have the Nagios server talk to that port -- unfortunately, the ISP's policy does not allow non-standard ports (namely any other than ssh-web) behind that server. Well, the ssh local forwarding technique proved to be the swiss army knife for us!

    Setting up the ssh tunnel

    We setup a local ssh forwarding tunnel from Nagios server, to the Nagios client machine. Then we configure the server to connect to that port, instead of the client:5666 port.The tunnel can be created by any user, as long as we use non-priviledged ports (> 1024).

    We first create a key pair on the server and copy the public key on the client, in .ssh/authorized_keys

    user@user:~$ ssh-keygen 
    Generating public/private rsa key pair.
    ...

    Then, we create the tunnel on the server side that installs an ssh tunnel: localhost port 34345 forwards traffic on client port 5666, as our client denies direct access to port 5666.

    nohup  ssh -L 34345:localhost:5666 nagios_client.net -N -i /home/user/.ssh/nagios_key &

    Notice the use of nohup; we want the connection to be kept on the background, and ssh -N which is helpful when forwarding ports, denying the execution of any remote command.

    Setting Nagios to use the tunnel

    Now on our nagios server we are reay to setup the client, as we have already done with all other servers

    define host{
    use                     generic-host
    host_name               nagios_client.net
    alias                   nagios client X
    address                 X.Y.Z.A
    check_command           check-host-alive
    max_check_attempts      5
    notification_options    d,u,r
    contact_groups          nagios-admins-two
    }

    On file commands.cfg, we setup a version of the check_nrpe command that will check for localhost port 34345 (the ssh tunnel) instead of the client's ip at port 5666, that is firewalled. This is the most important thing to remember, after setting up correctly the tunnel. We'll name this command as check_nrpe_firewalled

    define command{
    command_name check_nrpe_firewalled
    command_line $USER1$/check_nrpe -H 127.0.0.1 -p 34345 -t 100 -c $ARG1$
    }

    That will do the job. Any service that we need to specify for that client, we will have to prepend the nagios check with check_nrpe_firewalled.

    define service{
    use                             unweb
    host_name                       nagios_client.net
    contact_groups                  unweb-admins
    service_description             Check mail Queue
    check_command                   check_nrpe_firewalled!check_mailq
    }


    Once we create the host, set the services and install the Nagios client, we restart Nagios server and everything runs like a charm! We have successfully setup Nagios on our firewalled client, using the friendly help of ssh tunnels.

    This method also applies to other cases e.g. the client is behind a NAT (home servers, private data centers, etc). In this case however, instead of local ssh forwarding, we would need remote ssh forwarding.

    It's also useful for encrypting nagios traffic if you don't want to use the native SSL support Nagios >2.0 provides!

    Plumi 4.3 video platform released

    Featuring HTML5 video tag support with seamless flash fallback, WebM & MP4 transcoding, Universal Subtitles integration, 16:9 playback and more

    EngageMedia and Unweb.me are proud to announce the latest release of Plumi, the free software video sharing platform built on Plone. Plumi is a full featured CMS that enables the creation of video portals. Users can upload videos which will be automatically transcoded to web friendly formats for online viewing. Videos can also be uploaded through FTP, which is handy in case of big files. There are no restrictions regarding size/quality of the video. Plumi supports distributed video sharing via BitTorrent, subtitle creation and display through Universal Subtitles.

    Version 4.3

    Plumi 4.3 focuses on web standards, providing HTML5 video tag support with seamless flash fallback using MediaElement.js. It supports 16:9 video, adds a video language field and provides downloading links for all transcoded video versions.

    Plumi 4.3 provides on-line, user-generated subtitling support trough Universal Subtitles and comes with many bug fixes and improvements. The transcoding daemon is more stable than ever, and displays video transcoding progress to site managers.

    For an exhaustive list of changes and bugfixes, you can visit the following changelogs:

     

    More info

    We know you can't wait, so here's the link to the core package: http://pypi.python.org/pypi/plumi.app

    There is also a live demo system at http://demo.plumi.org for a quick glance of the product.

    Plumi comes with 2 installation options. One for production environments, which configures caching, multiple load balanced zope instances, etc and a simpler one for testing/development. Check out the installation docs on doc/INSTALL.txt and enjoy!

    For more info checkout the Plumi home page and the Plumi development site.

    Engagemedia.org is sponsoring Plumi development. Testing and bug reporting is more than welcome. As always, let us know if you find any problems via the contact form, or the email list. You can submit bugs and track development progress on the Plumi trac.

    Monthly archives on Django

    Build your own collapsing monthly archive for your Django project, in a few simple steps.

    Monthly archiving is a pretty cool feature offered by many blogs/sites. We wanted to add such a feature on the greek music portal liveinspector.gr, and specifically on the festivals page. We have created a simple solution that you can checkout on www.liveinspector.gr/festivals, and through the following article would like to share with the Django community. We would love to see the archive code on any Django powered blog/cms, and we have packed the example code on a small Django application that you can checkout on https://github.com/unweb/simple-archive.

     

    liveinspector.gr with the monthly archive for festivals on the right side

     

    Our example model is a simple model, with a DateField called date. This can be adjusted to whatever DateField field your models might contain.

    models.py:

    from django.db import models
    import datetime
    
    class Event(models.Model):
        "A very basic model for Events"
        name = models.CharField('A name for the event', max_length=200)
        place = models.CharField('Location where the event takes place',
                                 max_length=200)
        date = models.DateField('When the event takes place')
    

    On the view that we want to create the monthly archives now. First we make a query asking for all the events. Then we create a dictionary with years as keys, and for values we create dictionaries with months and the events for these months. Since in Python we can't create a dictionary with sorted keys, we create a list with dictionaries of years:months/events and pass it to our template.

    views.py:

    # Create your views here.
    
    from django.template import Context, loader
    from django.http import HttpResponse
    import datetime
     
    from models import Event
     
    def events_index(request):
        '''a basic events listing view'''
        events = Event.objects.filter().order_by('-date')
        now = datetime.datetime.now()
     
        #create a dict with the years and months:events 
        event_dict = {}
        for i in range(events[0].date.year, events[len(events)-1].date.year-1, -1):
            event_dict[i] = {}
            for month in range(1,13):
                event_dict[i][month] = []
        for event in events:
            event_dict[event.date.year][event.date.month].append(event)
     
        #this is necessary for the years to be sorted
        event_sorted_keys = list(reversed(sorted(event_dict.keys())))
        list_events = []
        for key in event_sorted_keys:
            adict = {key:event_dict[key]}
            list_events.append(adict)
     
        t = loader.get_template('templates/event_index.html')
        c = Context({
           'now': now,'list_events':list_events,
        })
        return HttpResponse(t.render(c))
    

    Finally, we pass the list_events list to our template. In our example, we want to show a monthly archive of the events, where months expand once clicked and show the events that take place on that month, with details as the location. Note that the current year/month is expanded automatically, through the variable "now" that we pass to the template. With the use of jQuery and css, we can present the data passed by the view to our template, the way we want.

    templates/event_index.html:

    <html>
    
    <head>
        <link rel="stylesheet" type="text/css" href="../static/main.css" />
        <script type="text/javascript" src="../static/jquery.min.js"></script>
    </head>
    
    <body>
        {% for event_year in list_events %}
            <ul class="year">{{event_year.keys.0}}
                {% for month, events in event_year.values.0.items %}
                    <!--show only months with events! -->
                    {% if events %} 
                        <li id="{{event_year.keys.0}}-{{month}}" class="month">
                            <div class="month-data">
                                <div class="collapsed">&nbsp;</div>
                                <div class="expanded" style="display:none;">&nbsp;</div>
                                <span class="name">{{events.0.date|date:"F"}}</span>
                                <span class="counter"> ({{events|length}})</span>
                            </div>
                            <ul class="events" style="display:none;">
                            {% for event in events %}
                                <li class="event">
                                <a class="title" href="#">{{event.name}}</a>
                                <div class="date">{{event.date|date:"j M Y"}}</div>
                                <div class="place">{{event.place}}</div> 
                                </li>
                            {% endfor %}
                            </ul>
                        </li>
                    {% endif %}
                </li>
                {% endfor %}
            </ul>
        {% endfor %}
    </body>
    
    <script>
        // toggle month
        var toggleMonth = function(month) {
            $(month).children('.events').slideToggle('slow');
            $(month).find('.month-data div.collapsed').toggle();
            $(month).find('.month-data div.expanded').toggle(); 
        }
    
        // expand the current month
        toggleMonth($('#{{now.year}}-{{now.month}}'));
        
        // intercept click to toggle month
        $('.month-data').click(function() {
            var month = $(this).parent();
            toggleMonth(month);
        });
    </script>
    
    </html>
    

    You are now ready to populate your database with some events and use your new archive. With the correct css and styling you will manage to achieve your desired results!

    Django simple-archive app

    As mentioned above, we have packed the code and created a small app, that can be used to test the archive solution. The installation below takes place with the help of virtualenv:

    First create a virtual python instance and install Django. Then clone the repository

    user@user:~/simple-archive$ virtualenv --python=python2.6 --no-site-packages ./django-test
    user@user:~/simple-archive$ cd django-test/
    user@user:~/simple-archive/django-test$ ./bin/pip install django
    user@user:~/simple-archive/django-test$ git clone http://github.com/unweb/simple-archive.git simple-archive
    

    sync the db, and load some fixture data

    user@user:~/simple-archive/django-test$ cd simple-archive/
    user@user:~/simple-archive/django-test/simple-archive$ ../bin/python manage.py syncdb
    user@user:~/simple-archive/django-test/simple-archive$ ../bin/python manage.py loaddata fixtures
    Installed 10 object(s) from 1 fixture(s)
    user@user:~/simple-archive/django-test/simple-archive$ ../bin/python manage.py runserver
    Validating models...
    
    0 errors found
    Django version 1.3.1, using settings 'simple-archive.settings'
    Development server is running at http://127.0.0.1:8000/
    Quit the server with CONTROL-C.
    

    Now point your browser to 127.0.0.1:8000 and you'll see the archive, with fictitious data.

    Example of a monthly archive expanded

    Example of a monthly archive collapsed

    How to add a Reply-To header in email messages sent through Plone

    Plone ships with an author feedback form that enables logged in users to contact other portal members. When the form is submitted, an email message is sent to the target user, using the site's global email address in the "From:" email header. We had to partially change this behavior by adding a "Reply-To:" in the message headers, so that when the user replies to the email received, the reply will be sent to the user that filled in the contact form, rather than the Plone site's global address.

    The solution is easy, albeit not immediately obvious. All we need to do is customize author_feedback_template.pt and prepend the following line.

    Reply-To: <span tal:replace="options/send_from_address"/>

    We can do that either through the web using the ZMI, by pointing our browser to http://mysite.com/portal_skins/plone_templates/author_feedback_template/manage_main and clicking on customize, or through the file system, by adding a modified author_feedback_template.pt in the myproduct/skins/myproduct_custom_templates folder of our custom Plone product.

    Workshop: φτιάξε το δικό σου youtube με ΕΛΛΑΚ!

    Την παρασκευή 20 Μαΐου στα πλαίσια του συνεδρίου ΕΛ/ΛΑΚ 2011, θα πραγματοποιήσουμε workshop εισαγωγής στην πλατφόρμα δημιουργίας video portals Plumi.

    Στο πολύ ενδιαφέρον πρόγραμμα του φετινού συνεδρίου ΕΛ/ΛΑΚ εντάσσεται και το workshop για το Plumi. Αν βρίσκεστε στην Αθήνα την Παρασκευή 20 Μαΐου, ελάτε μια βόλτα από τις Αίθουσες Βιβλιοθήκης της Πολυτεχνειούπολης Ζωγράφου (οδηγίες πρόσβασης) μαζί με το laptop σας για να εγκαταστήσουμε και να τροποποιήσουμε το Plumi. Στόχος, στο τέλος του workshop, είναι να έχουμε το δικό μας video portal à la youtube, με χρήση ελεύθερου λογισμικού! Το workshop θα ξεκινήσεις στις 12.00 και θα διαρκέσει περίπου δύο ώρες.

    Το πλήρες κείμενο της παρουσίασης μπορείτε να το βρείτε σε αυτό το link.

    Plumi 4.0 video platform released

    Plumi 4 is out, better and faster!

    We are proud to announce the latest release of Plumi, version 4.0, the free software video sharing platform built on Plone. Plumi is a full featured CMS that enables the creation of video portals. Users can upload videos which will be automatically transcoded to web friendly formats for online viewing. Portal adm

    inistrators may decide if a video will be published, featured on the front page, or rejected. Videos can also be uploaded through FTP, which is handy in case of big files. There are no restrictions regarding size/quality of the video.

    Version 4.0

    Plumi 4.0 is based on Plone 4.0, which is a lot faster compared to both previous Plone versions as well as most other popular content management systems.

    Plumi 4.0 comes with 'batteries included'. Everything you need in order to set up a high traffic video portal is automatically installed by the production buildout. That includes up to 8 Zope instances, ZODB obect database on a ZEO setup, Haproxy load balancer, Varnish cache server, Transcode Daemon for video transcoding, PloneFTP for ftp uploads, NgInx web server and a Supervisor to manage all those processes.

    Plumi 4.0 has less fat. Some of the less vital features of Plumi, like taxonomies and bookmarklets, have been factored out as optional components, in order to make Plumi more suitable for generic use cases.

    A lot of bugs have been fixed since the previous release. The release candidate has been thoroughly tested on production at engagemedia.org and is free of major issues.

    Stay tuned with Plumi development. Coming versions will provide BitTorrent downloads, video analytics integration using Piwik, better support for mobile devices and much more.

    More info

    We know that you can't wait, so here's the link to the core package: http://pypi.python.org/pypi/plumi.app

    Plumi comes with 2 installation options. One for production environments, which configures caching, multiple load balanced zope instances, etc and a simpler one for testing/development. Check out the installation docs on doc/INSTALL.txt and enjoy!

    For more info checkout the Plumi home page and the Plumi development site.

    Engagemedia.org is sponsoring Plumi developmentTesting and bug reporting is more than welcome. Let us know if you find any problems via the contact form, or the email list.

    Introducing collective.piwik.*

    A new set of tools for integrating advanced analytics to Plone sites.

    At the end of this year's Plone conference in Bristol, we took part in the 2-day Plone video sprint. We have worked along with Giorgos Logiotatidis and the result was the creation of collective.piwik.*, a set of Plone products that provide analytics content into Plone sites using Piwik.

    Analytics support is a necessity for every modern web site. We use the great open source system Piwik and need a set of tools that can be integrated in Plone sites to display data such as number of views of each page, number of visitors currently using the site, number of times a video has been played, etc.

    Piwik provides an API that exposes data on several formats, including XML and JSON. We decided to fetch the data from a Piwik server, and display it on viewlets or portlets, that can be put anywhere. We didn't want to create one package with all the viewlets, because this would be unnecessary for most Plone sites, but instead create separate packages for each of the viewlets. Thus we created:

    The above products are still on alpha releases and more work is planned during the coming month. Extra effort will be given to assure there are no security and privacy issues. At the moment the user's browser queries the Piwik server directly, so the Piwik server has to be configured to provide the analytics data to anybody, which is a not wanted dead in most cases. The architecture will soon be changed so that the piwik API is invoked by the Zope server in order not to expose any data that shouldn't be public.

    The products will soon be integrated on engagemedia.org. Thanks to anyone who tests and reports bugs!