Following our previous article, with the sudden subscription of hundred of new users both on our XMPP #services movim.eu and jappix.com and on #Movim itself it seems that we are starting to be quite limited by our current servers capacity.

You might have noticed some connection issues and downtimes the past few days that are the direct result in this recent gain of interest for Movim.

In the upcoming days we are planning to migrate the whole #infrastructure to a new server to give a bit more space for the project to grow and to allow many new users to join us.

This will involve some extra server costs. If you want to help us covering our expenses you can always join our Patreon.

By the way, we also bumped the size limit from 3Mb to 10Mb on our XMPP upload service, enjoy!

So it seems that we doubled our daily registrations the past few days 🎉 .

Thanks to #WhatsApp lots of new people are discovering XMPP and its ecosystem . We also recently rolled-out an improved and simplified flow for the registration process. You can find it there Register on movim.eu and share the link around.

We will also communicate soon about some big news about #Movim itself, stay tuned 😉

A lot of new exciting features were added to #Movim the past few weeks! Like Chat Reply, that allows you to quickly quote and reply to an exisiting message in a one to one or chatroom discussion.

It is now also possible to search and add GIFs within the discussions thanks to the Tenor integration. This feature is totally optional. The administrator can easily enable it by adding the #Tenor API key within the Movim admin panel (don't forget to restart the daemon).

The embedding feature was also greatly improved. Movim now resolves video urls, pictures as well as general website URLs to embed them properly within the chat! Some improvements in the picture preview feature now also allows you to preview pictures from the embedded url quickly before visiting it.

And finally, some improvements were made to the picture proxy feature. To simplify, Movim is protecting its users by serving the externally shared pictures (from Internet websites for example) and acting as an intermediary. It also automatically recompress the pictures if they are too large.

Server administrators that are hosting a Movim pod can now easily setup a cache system allowing those Movim proxyfied pictures to be kept for a while and reduce the load on the Movim side. For example, if someone is sharing a large picture URL in a chatroom, the first time Movim will display it to a member, it will recompress it and keep it in a cache, the other users will then directly request the cached version and load it instantly.

The One-Page-Setup wiki page has been updated to explain how to setup that cache easily.

With all those new features Movim is now ready to compete with other web chat platforms such as #WhatsApp or #Discord. But with the power of decentralisation, standard and build on open-source technologies.

It also seems that a big feature might be planned for 2021, stay tuned ;)

If you enjoy Movim and want to help funding the project, you can help us on Patreon. This will allow us to cover our monthly expenses and fund some new features.

Thanks again for your support!

That's all folks !

A few days ago I discovered that the #Eloquent database library (the one that Movim is using for a few years now) had some events that could be caught by an event listener.

With this simple mechanism #Movim could be aware of when exactly a #SQL request is done during the execution time.

The Movim #architecture is based on several processes that talk to each others, and each connected session (each user connected to a specific XMPP account) is having its own sub-process. The main issue with this architecture is that all those processes relies on a common database (PostgreSQL or MySQL) and each process open an unique connection to the database. See How's Movim made? Part I - The Architecture to know more about this architecture.

When you start to have a lot of users connected on the same instance, this is opening lots of connections to the database. This is creating a congestion and can slow down the general performances.

This simple commit ask Eloquent to close the connection after a few seconds of inactivity. Eloquent is reconnecting automatically if a new SQL request is made after the disconnection. This allows to free-up the socket resources once "we don't really need it anymore".

And here is the result once this commit deployed on nl.movim.eu

It's been six months since the last #release, and for the 0.18 – Oterma one there's a lot to show!

So let's dive in this list of exciting features and changes that you will find in this new release.

Notifications

0.18 is coming with quite some changes regarding how the notifications are handled and displayed within Movim. This also came with lots of bug fixes to ensure that the counters are up-to-date everywhere across the UI and between all the devices.

One of the first notable changes you will find is how the global chat counter is behaving. This counter is now displaying the total number of unread discussions and not the total number of unread chat messages anymore.

Related to this change, an update on the chatrooms bookmark specifications is now allowing you to save a per-chatroom notification configuration. With this new feature you'll be able to choose, for each chatroom, if you want to “always be notified” for each new messages, “only when you're mentioned”, or “never”.

Movim can now use sound to notify you of a new incoming message, if you allow it to (see the configuration panel).

Stickers, drawing and emojis

We also improved some existing chat features. Some work has been done by Christine Ho on the Draw widget. The drawn lines are now smoother, even on large resolution pictures. The feature has also been extended to allow you to draw on existing pictures you upload from your device.

Christine Ho also improved the existing emoji picker to add useful search bar, you can find it when trying to add a reaction bellow a message.

A new pack of fresh meme stickers was also added to the stickers panel.

And finally, the Movim emojis support has been updated to Unicode 12.0, bringing a pack of new emojis in.

Video conferencing

A big part of the time spent on this release was focused on the video-conference features. The related code was heavily reworked to ensure a smooth compatibility with the #Conversations Android client, it's still not perfect but works already quite well.

This was especially because of the integration of XEP-0215: External Service Discovery in the project. Allowing the web browser to easily perform STUN/TURN discovery (it's a way to discover alternative routes on the Internet for the video and audio feeds to be streamed on) using the #XMPP server services.

The call flow was also split to conveniently allow audio-only calls.

And the screen-sharing feature was integrated on the video-side. Useful when you want to quickly show a document or a presentation during your call!

Video-conferencing is hard. For this release we made quite a lot of work to improve the user experience but we are still not totally satisfied with the result. Some more work will be needed in the upcoming releases to really deliver the best we can offer for that feature.

And a many other things…

Chat and chatrooms

The interfaces between the chat and chatrooms (header bar, drawer menu) are now unified to display general information the same way. A little gallery was added which displays recently sent and received pictures.

The pictures handling in chat messages has been improved. Movim is now trying to resolve picture URLs, even if they were sent using a client that doesn't embed pictures in messages.

The scrolling behavior in chat was greatly improved and simplified, a little "go to bottom" button was also added.

Movim now handles the XEP-0319: Last User Interaction and can therefore be more precise to tell you the last time your contacts were seen active.

Post publication

Movim was already automatically saving your draft posts while you were writing, but since this version it actually displays when a draft has been saved. The post publication panel was also slightly redesigned to be clearer and more accessible.

Performance

On the technical side, a PresenceBuffer system was added to handle wave of presences coming from the network and mass insert them in the SQL DB. This reduces by a factor of 10 to 100 the number of requests that are done to the database during those "spikes" of incoming presences from the network. This little feature greatly improves logins speed and shortens the time it takes to join a new chatroom.

Favicon

The favico.js library, that was handling the little counter in the browser tab icons had been removed and rewritten using only vanilla JavaScript, keeping only the required bits which also made it way smaller. This was actually the last (external) JavaScript dependency coming with Movim.

XMPP Authentication

Regarding the authentication process, Movim now supports XEP-0368: SRV records for XMPP over TLS to directly connect to XMPP using an encrypted socket (better than the historical STARTTLS negociation system that is still used as a fallback mechanism). Regarding the IP resolution Movim now uses the Happy Eyeballs mechanism to resolve the server IP on IPv4 or IPv6.

Android application

Finally, the official Android app gained nice features thanks to the improved integration of Movim on your phone.

The notification handler has been reworked to handle newer Android versions, conversations are now stacked properly.

From any application you can now share a URL to Movim directly and publish it in a Post. In the future we are planning to add more sharing capabilities like this one (share to contacts or chatrooms for example).

Related to the video-conferencing improvements, audio and video calls are now launched in a separate task to allow you to switch back easily to your chat or publications while having your call. This makes it close to the experience you can already have on desktop.

Finally some work was done in #Movim and on the Android app to improve the general performance, especially regarding the touch features and general responsiveness.

To conclude…

As always you can test all those features on our official pod, or deploy them in a few steps on your own server :)

We always need some funding to help us promoting Movim and covering project costs, feel free to help on our Patreon.

And stay tuned for the upcoming releases, it seems that there are some interesting features already planned ;)

That's all folks!

And also here

Recommend Movim and share your experience ;-)

(No registrations recquired).

#Movim, #XMPP, #Recommendations

The movim.eu #XMPP server now supports connections through #WebSockets.

This means that if you're using a #web client like conversejs, you'll be able to connect directly using your movim.eu account.

If you are a #Movim user. Nothing changes as Movim is using standard good ol' TCP/TLS connections to login 😉

The clients that wants to connect to movim.eu using WebSockets will normally automatically discovers the endpoint (using the related XEPs). If not, it is available manualy at wss://movim.eu/xmpp.

That's all folks!

Movim 0.18 is planned to be released soon.

In the meantime, let's have a look at one specific feature that is really useful when you integrate Movim with the Spectrum2 - Telegram bridge.

For those that are not aware, XMPP can connect to other chat networks using tools called "transport". One of the most used is called Spectrum2 and can connect to many other networks thanks to its libpurple support.

Telegram transport setup

What will we do here is:

• Setup telegram-purple in Spectrum2 on Debian
• Connect it to a XMPP server (here ejabberd)
• Adapt the transport to integrate with Movim

Setup Spectrum2 and telegram-purple

Here I will not detail the basic installation, the official Spectrum2 documentation is pretty complete.

Once the repository is setup, please install the base package and the libpurple module:

apt install spectrum2 spectrum2-backend-libpurple

For telegram-purple the README is also quite complete

Create a Telegram transport

Once all the packages are setup, we will create a transport configuration file. You can reuse the spectrum.cfg.example located in the /etc/spectrum2/transports/ as a base.

# nano /etc/spectrum2/transports/spectrum_telegram.cfg

This is basically the config file that I used for my own telegram.movim.eu transport:

[service]
server_mode = 0
user=spectrum

jid = telegram.movim.eu
password = spectrumpassword
server = 127.0.0.1
port = 5347
backend_host = 127.0.0.1

users_per_backend=10

backend=/usr/bin/spectrum2_libpurple_backend
protocol=prpl-telegram

web_directory=/home/movim/upload/spectrum
web_url=https://upload.movim.eu/spectrum

[identity]
name=Telegram Transport
type=telegram

[logging]
config = /etc/spectrum2/logging.cfg
backend_config = /etc/spectrum2/backend-logging.cfg

[database]
type = sqlite3

[registration]
enable_public_registration=1

When Spectrum2 will connect to the Telegram network, the stickers will be downloaded as files on the server. By default an ugly path is simply sent to the XMPP clients. We will turn it to a proper URL and let Movim to its magic.

To do that we need to configure the Web Storage module. You can also find more documentation about it there.

web_directory=/home/movim/upload/spectrum
web_url=https://upload.movim.eu/spectrum

It's pretty self explanatory. The downloaded stickers will be put in the web_directory directory. The second parameter, web_url, is basically telling Spectrum2 how to general its URL before sending them in the messages.

Configure ejabberd

Then we need to add a new service in our ejabberd.yml configuration file.

  -
    port: 5347
    module: ejabberd_service
    access: all
    ip: "127.0.0.1"
    global_routes: false
    hosts:
      "telegram.movim.eu":
        password: "spectrumpassword"

Once everything is setup, restart Spectrum2 and ejabberd. For Spectrum2 you can do it using spectrum2_manager or a dedicated systemd configuration file.

Configure our web server

We then need to expose those files to the web. A simple nginx configuration will handle it.

server {
    server_name upload.movim.eu;

    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    …

    root /home/movim/upload;

    location /spectrum {
        alias /home/movim/movim/spectrum;
    }
}

Fix the nasty file rights with Incron

If you start to use your Telegram transport at this point you'll notice that the stickers URLs are returning a 403 Forbidden error.

Indeed, Spectrum2 is writting the files in the directory using it's own rights. And this can't be configured.

We will then use another useful tool called Incron. This tool works like CRON but instead of working on time events, it works on file events.

You can find a pretty complete documentation there.

apt install incron 
nano /etc/incron.allow # add your spectrum user there
sudo -su spectrum
incrontab -e

In the incrontab file well then change dynamicaly the rights of the files once they are wrote in the directory (check the documentation for more details).

/home/movim/upload/spectrum   IN_CLOSE_WRITE          chmod 664 $@/$#

Enjoy your nice Telegram stickers in Movim

In Movim, nothing more to do. With the version 0.18, Movim will try to resolve the incoming messages that contains a URL and see if it's a valid picture. Which is the case for Telegram stickers.

You can also note that it works for any other incoming picture URL, including those sent using Conversations or other XMPP clients.

That's all folks!

#telegram #xmpp #movim #transport #stickers #ejabberd #admin