Quick Project: Mouse Coordinate Image Mapper

As a rather quick project and something to keep my free time, I decided to mess around with the Windows User32.dll functions.

Digging through I found two functions that caught my interest instantly:

mouse_event(int dwFlags, int dx, int dy, int dwData, int dwExtraInfo);

and

SetCursorPos(int x, int y);

Both of those functions control the position and events applied to the user’s mouse. Interestingly enough, this allows applications to fully control the mouse, which leads me to assume this is what most “auto-clicker” applications tap into to enable the trivial functionality they offer. Continuing on with my programming, I decided it would be rather interesting to try and implement a two-tone filter using only the mouse.

By using this handy widget in Opera, I measured the X and Y Cartesian distances from the top-left edge of the screen to Microsoft Paint’s canvas when aero-snapped to the left. After that, I left the distances as constants in the source code that can be altered for any other program. With Paint measured out, I was able to write a simple goto and click function which is used in the algorithm when a single pixel’s RGB value summed is less than 127 * 3; from there if the previous assertion is correct it will click that pixel’s location plus the buffer space to the canvas as previously measured.

Thus the following code was produced to carry out this operation:

 class Controller
    {
        public static bool InProcess = false;
        public static bool RemoteStop = false;
 
        const int PAINT_FROMTOP = 150;
        const int PAINT_FRONLEFT = 100;
 
        [DllImport("user32.dll", SetLastError = true)]
        static extern void mouse_event(int dwFlags, int dx, int dy, int dwData, int dwExtraInfo);
 
        [DllImport("user32.dll", SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        public static extern bool SetCursorPos(int x, int y);
 
        [Flags]
        public enum MouseEventFlags
        {
            LEFTDOWN = 0x00000002,
            LEFTUP = 0x00000004,
            MIDDLEDOWN = 0x00000020,
            MIDDLEUP = 0x00000040,
            MOVE = 0x00000001,
            ABSOLUTE = 0x00008000,
            RIGHTDOWN = 0x00000008,
            RIGHTUP = 0x00000010
        }
 
        public static void LeftClick(int x, int y)
        {
            SetCursorPos(x, y);
            mouse_event((int)(MouseEventFlags.LEFTDOWN), 0, 0, 0, 0);
            mouse_event((int)(MouseEventFlags.LEFTUP), 0, 0, 0, 0);
        }
 
        public static void DoPaint(Bitmap image)
        {
            InProcess = true;
            int currentPosX = 1;
            int currentPosY = 1;
 
            do
            {
                if (RemoteStop) break;
                Color cTemp = image.GetPixel(currentPosX, currentPosY);
 
                if ((cTemp.R + cTemp.G + cTemp.B) < (127 * 3))
                    LeftClick((currentPosX + Controller.PAINT_FRONLEFT), (currentPosY + Controller.PAINT_FROMTOP));
                else continue;
 
                if ((currentPosX + 1) >= image.Width)
                {
                    currentPosY++;
                    currentPosX = 1;
                }
                else
                {
                    if ((currentPosY + 1) >= image.Height) break;
                    else currentPosX++;
                }
 
                System.Threading.Thread.Sleep(2);
            } while (true);
 
            InProcess = false;
            RemoteStop = false;
        }
    }

To top off the project I created a simple GUI to proxy the actual function. There is, unfortunately, a need for System.Threading.Thread.Sleep(2). The code runs faster than Paint can keep up with, thus the code needs a slight handicap to ensure Paint can successfully keep up with the quick drawing by many clicks.

If you neglect handicapping the program, you’d receive this as a result.

After several failed attempts and quite a bit of code revisions, I finally achieved what I aimed for… capped at 2 milliseconds per dark pixel, ignoring light pixels, and utilizing a keyboard hook. You can see the video of it in action here. Although it was slow in the video, the new code is revised to be much faster; the video just serves as a functionality demonstration.

My plans for expanding this involve OpenCV. Perhaps reading human movements into mouse movements would make quite an interesting project.