You can open Mesh Gradients in Krita now!

TL;DR: meshgradient get rendered in Krita just like you'd expect them to render in other apps

Well, because I couldn't get Bicubic interpolation fully working by the time of writing and publishing this blog. This part is still in pending state :(

Firstly, here are is a screenshot of a complex meshgradient which I found from Inkscape (I modified it to be Bilinear to get this):

pepper.svg



As, there isn't much else to say besides giving out the details, I'll jump straight to the technicalities :-)

Technicalities

Rendering (Bilinear)

I started with reading the algorithm mentioned in the specs sheet i.e Divide And Conquer. I had some confusions about it, but thanks to my mentors. I got the idea. Then, to implement this algorithm, the idea was to subdivide the patches, until the difference between the corners is small enough.

Now the question was, how am I going to divide the surface. Luckily, there was a paper, one search away, which I picked up. I read it quickly and started writing the algorithm on paper. Somethings didn't quite make sense, but as I later found out that they were probably misprints.

So, as I implemented the algorithm, the first thing that I tried to open was this, to check if the subdivision algorithm was right. Comparing this to Inkscape, it was pretty accurate (not the color, but the surface itself).



Next, get colors for the new corners was simple too. You could get it by dividing it in half (we get this from the fact that the surface is Bilinear). Then just mix them when difference is less than the tolerance and you get this:


Caching

Because rendering meshgradients is expensive and anything e.g a simple translation would call for a complete redraw. I had to write a way for poor man's caching. Instead of painting directly on a the canvas. I would first paint on QImage and then paint it on QPainter.

Simple enough! However, there was a problem, rendering on QImage and then on scaled up QPainter made things look very pixelated because of scaling. So, to counteract this, my solution was to paint on scaled up QImage i.e scale it up by the same factor as the QPainter. This made everything a whole lot better.

Rendering (Bicubic Interpolation)

This has been the hardest part about this GSoC (till now). The way I usually write code is, I write everything in a bit messy way till I get something running. Then I come back and check if something could be written in a better way (read efficiency and readability). This approach went horribly wrong as I probably had made mistakes and my render was pure chaos and nowhere close to what you call interpolation. Then I git stashed everything and started over in a more manageable way, now the interpolation was working. But turns out, it was an exact replica of the Bilinear interpolation, but slower (because it did actually calculate derivatives and what not).

So, I asked for help. But then I immediately realized that my assumptions were wrong. In short, I was calculating derivatives incorrectly. I quickly modified the code in the dirty way and finally I could see a smooth render with no Mach Banding. Unfortunately, this too is based on an assumption, which is not correct for the all cases. So, Bicubic interpolation works neatly for Mesh Patches which are relatively linear. But falls out of place for a more messy (generic case) ones.
But why? The problem here is with the way I do subdivision, it is only valid for linear Coons patches. I haven't written the code to subdivide a Bicubic surface. That isn't the only problem, subdividing Bicubic surface is an expensive operation as well. So, I'll have to find a middle ground.


Saving

Since I spent a lot of time on getting Bicubic Interpolation to work, for a context switch I moved to the next milestone.

So, I tried to implement the saving operation. Thanks to all the abstractions! This was pretty straightforward to implement.

I will now write tests for it and the final rendering :)

That's all I have for now, hopefully I'll post an update when I get the Bicubic shading working :)

Bugs

...while rendering

Hello once again!

First of all, sorry for not making a blog post early on during the community bonding period. I couldn't because I was mostly busy with Krita's Android release.

Secondly, some of you might remember me from the previous year. I was GSoC student for Krita. Now it is my second time! :-)

Finally, to tell a bit about my project. My project is to add support for SVG mesh gradients in Krita and the relevant task is: https://phabricator.kde.org/T13101 and branch is: https://invent.kde.org/graphics/krita/-/merge_requests/378

Now over on to the interesting stuff!

So, let's see what I did the past couple of weeks.

1. Parsing

This is probably going to be easiest and most of us know hows and whats about his. So, we throw any SVG with meshgradient element, in it and Krita understands it now.

2. Making a shape out of the path.

Now, this turned out to be a bit complicated than I had anticipated, not because this was hard. But because there were a few edge cases which I overlooked. I would fix one edge case, but it would break the other one, classic whack-a-mole. So, I had to do a few rewrites of this tiny 'SvgMeshPatch and SvgMeshArray' component. Finally, I just put it all on paper and got it working for all cases, in a proper manner *.

The way I started was to manually compare some 'edge case' values to see if they're correct and fit the logic. But, there's so much that an eye can overlook. What I think I should've done is to write unit tests and then handle each edge case respectively. However the good thing is after writing unit tests, I discovered two cases where my logic wasn't right.

3. Deeper dive into the technicalities

I will try to explain how I did in a bit more detail. If you have any suggestions/critique, you're welcome.

In the big picture sense there are two things to consider, when talking about meshgradients, meshpatch and meshrow (which is a linear array of meshpatches). As per the specifications, meshpatch is a Coons Patch, which is just the shading/fill defined by the interpolation of colors placed on the corners (i.e. edges of the curve).

Because they can be seen as a two dimensional arrays, creating an array of meshpatches, seems the most logical approach and that's what I did. However there is a slight catch, each patch shares side with other patches. Eg. in case of a gradient with a single row, up to two sides are shared. So, while parsing this had to be taken care of.

So, now we have SvgMeshArray which is an array of of SvgMeshPatch and each SvgMeshPatch owns a KoPathShape, which is how the fill boundaries are going to be defined. And that's basically all there's to it for now...

On a side note, one thing which is in my mind is that because each meshpatch is treated as an individual Shape, there's some duplication with the shared sides. But I think this is an optimization problem, which has to be taken care of, but after rendering :-)

Now a couple of obligatory screenshots, just to double check :)





PS:

* I in no way consider myself an expert, yet. Feel free to look at the code and comment on it.

Updates on Krita on android

Update

It has been a long time since I posted a blog (1 month+). In fact it might even seem the Krita ran on Android and now GSoC is done. Well, not quite. There’s still a lot to be done.

Let’s see what we worked on :)

Build

First and foremost was managing the build. If you have a look at README.android, you will see that’s a bunch of environment variables and few steps. If you look at underlying code, it was even uglier with lots of boiler plate. So, we had to refactor it and we did and made the build system a bit pleasant. (4467ad274)

OpenGL Canvas

Next, was a bug in the OpenGL canvas, when we enabled hardware acceleration, the canvas would turn black. QPainter being painfully slow, so it certainly isn’t the option, so we really rely on OpenGL ES.
So we had to fix this, it took me quite some time to fix and the solution was simple in the end.
Results? We can draw on OpenGL canvas now!

Autosave

If you have played any heavy game on Android, then there is one thing which you might know. If you leave the app and come back after a few minutes all the progress is lost or the connection to the server is terminated (it happens less often with new devices having a lot of memory).
This could happens with Krita as well, so now as soon as we get Activity#onPause(), we call the JNI wrapper Java_org_krita_android_JNIWrappers_saveState to save the state synchronously.
I did try to save it asynchronously (which uses slots and signal), but I found out, as soon as the main Qt thread was paused, the entire queued connection mechanism was brought to halt.

Touch events

For past week, I have been working on adding touch support to the canvas. What I mean by touch support is, to handle “finger paint” events and we’ve been successful in doing so. We can now draw on canvas now, using our fingers, not just a pen!

To do this we simply consider the touch events with one touch point as mouse click/update/release events (the same way we do this for tablet events).

But… there is a difference!

Krita did support touch drawing on touch screen windows/linux laptops. So, why didn’t it work for android devices? In windows, if QTouchEvent wasn’t handled, Qt would automatically generate QMouseEvent for it. But the same wasn’t true for android.
There is a way, however, to simply ignore touch events and use mouse events, even for android (by manipulating Qt::WA_AcceptTouchEvents flag). This wouldn’t work for Krita because we still use touch events to rotate/zoom/move canvas. So, we now explicitly handle touch events.

Rotation

We can now rotate the canvas using gestures! A small clip: rotation-canvas

It was pretty simple as well. First the way KisTouchShortcut::match worked, wouldn’t allow both zooming and rotation to co-exist, because it distinguishes the different KisTouchShortcuts based on number of minimum and maximum touch points. For both zooming and rotating, it was two.

So, for them to work in harmony, I created another class, KisZoomAndRotateAction, which delegates the call to KisZoomAction and KisRotateAction.

Secondly, in KisRotateAction, we just find out the angle between the lines. Line being, “the line” passing through the two touchpoints on the canvas.
So, the two lines are:

  1. Initial position of fingers
  2. Final position of fingers

(this is a bit hard to explain, please look at code).

Anything else?
Yes, this.

That's all! Thank you for reading, I’ll try to be more regular with my blog now :)

Debugging Krita on Android

 Debugging Krita on Android

Well, the easiest way is to use Android Studio.

Import the project in Android Studio as a gradle project and build the project. Krita build will fail when run from Android Studio. Now, to run it successfully, we’ll have to manually provide the path to installPrefix or comment out copyLibs dependency. Now, the project should build properly.

You might want to change the debug type to Native or Dual, as their Auto mode did not work for me. Open the C++ file in Android Studio and set a breakpoint. Click the bug icon, sit back and watch while Android Studio does all the magic for you.

And then it is usual lldb (in Android Studio) or GUI if that’s what you prefer.

Using command line:

Starting Android studio takes a lot of time and memory. Then, it builds which takes an additional few minutes, so it really isn’t a good idea to use it for debugging every time the app crashes. So, the less time consuming one and a bit complex method – here we go!

Assuming the app has been installed with the debug key. The first step is to launch it in debug mode, to do so:

# domain/(launcher activity or exported activity's class-path)
$ adb shell am start -D -n "org.kde.krita/org.krita.android.MainActivity"

Now the app on phone should launch and show Waiting for Debugger message. While it waits – open a terminal and enter $ adb shell, and then look for lldb-server in /data/local/tmp/. If you ever debugged app through Android Studio, then it should exist. If it does not, then launch Android Studio and run it in debug mode 😂…hahahaha.

Just kidding, push the file to that location.

$ adb push $ANDROID_SDK_ROOT/lldb/<version>/android/<abi>/lldb-server /data/local/tmp
$ adb shell chmod +x /data/local/tmp/lldb-server

(No lldb directory? See notes)

Then for us to access the libraries, we’ll have to copy it to /data/data/org.kde.krita, for that:

$ adb shell run-as org.kde.krita cp /data/local/tmp/lldb-server /data/data/org.kde.krita

(Why run-as? It is a setuid program and gives us the necessary permission to access the sandbox).

Now, enter the app sandbox by first entering the $ adb shell and then $ run-as org.kde.krita.

Run the lldb-server like you would if you were remote debugging.

$ ./lldb-server platform --server --listen "<incoming-ip>:<port>"
$ # Example: allow any ip on port 9999
$ ./lldb-server platform --server --listen "*:9999"

Now on the host machine, run lldb and then

(lldb) platform select remote-android
(lldb) platform connect connect://<ip>:<port>

On my machine:

(lldb) platform select remote-android
  Platform: remote-android
 Connected: no
(lldb) platform connect connect://localhost:9999
  Platform: remote-android
    Triple: arm-*-linux-android
OS Version: 28.0.0 (4.4.153-15659493)
    Kernel: #2 SMP PREEMPT Thu Apr 4 18:31:57 KST 2019
  Hostname: localhost
 Connected: yes
WorkingDir: /data/data/org.kde.krita

You can read more about what they do on LLVM’s website.
(This is one a time setup you can keep the server and client connected)

Remember, that our process is still Waiting for Debugger? :(
Let’s give it what it wants. Attach the debugger to the running process’s pid, which can be known by $ adb shell ps | grep "krita" or $ pgrep "krita"

To attach:

(lldb) attach <pid>
(lldb) # on my machine
(lldb) attach 1818
Process 1818 stopped
* thread #1, name = 'org.kde.krita', stop reason = signal SIGSTOP
    frame #0: 0xe8d35f7c libc.so`syscall + 28
<and much more>

Still didn’t continue? :-<
So, let’s finally resume it!

We’ll have to resume it over Java Debug Wire Protocol (JDWP), we’ll use jdb

$ adb forward tcp:12345 jdwp:<pid> # the same pid which we attached in lldb
$ jdb -attach localhost:12345

Now continue the process in lldb and we are done!

(This might seem like a lot, but it really isn’t. Every time the app crashes, I run in debug mode $ attach pid and I get the backtrace immediately!)

PS: When I was looking for it on the internet, I didn’t find much about it and had to spend a lot of time on this.
This method should work with debugging any android app with lldb, obv!
(I am really new to blogging. If it’s hard to understand or my formatting is bad, I am really sorry.)

Notes

  • I hate the extra jdb thing, and if the function which you want to debug is not going to be called during the early start up, you can use -N flag instead of -D with am.
  • Can’t find lldb directory in your SDK? Use platform tools to install it.
  • jdb doesn’t attach? $ killall android-studio && adb kill-server #_#

Resources

https://source.android.com/devices/tech/debug/gdb
https://android.googlesource.com/platform/ndk/+/master/ndk-gdb.py
https://lldb.llvm.org/use/remote.html

Hello KDE

Hello, my name is Sharaf. My nick on IRC is sh_zam.

My project is to port Krita to android devices. We've been successful in making the APK, but it only works if I build it, as it requires tweaking qt libraries, a bit. At the moment, my goal is to make the build system fully automatic and spit out the signed APKs for different architectures at the end.

Once I do that, I'll move on to UI, events and other fun stuff!

So, there's a lot to do and learn. Now I will go back to coding. (-:

So, thank you KDE for choosing me and I hope I'll learn a lot from this community!

You can open Mesh Gradients in Krita now!

TL;DR : meshgradient get rendered in Krita just like you'd expect them to render in other apps Well, because I couldn't get Bicubi...