Published by Tsehp

In my last essay I uncovered the secrets of FlexLM vendor defined encryption routines. These are pretty easily defeated. There is another more commonly used tactic called vendor defined

checkout filters. This is another way a vendor can protect their applications from generic cracks of FlexLM. This essay is intended to show what this added protection is about and how to defeat

it with ease.

Target:

• Any FlexLM protected target that implements vendor defined checkout filters. I've provided a sample application I built myself here vendor_chkout_filter.exe.
  

Tools:

• Softice - Debugger
• C Compiler
• WinDasm or IDA- Disassembler
• Flexlm Software Developers Kit (I'm using v6.0 for this essay)

Knowledge:

• I suggest, as a prerequisite, that you read the great essays on a +Fravia mirror by Nolan Blender, SiuL+Hacky, Pilgrim, Dan, and VoxQuietis.
• I also suggest that you read the documentation that comes with the FlexLM sdk.
• My previous essay on vendor defined encryption routines. Found here.

OK, lets start. I'm going to use a similar approach to tackle this as I did in me previous essay. First, I'll tell you that vendor defined checkout filters basically consists of extra parameters in the license file that are defined by a vendor. This can include many things like restricting the platform the target is run on to just about anything the vendor desires.  From my previous essay we've learned that FlexLM uses   a function called lc_set_attr to control the behavior of FlexLM itself. So once again we will start to look at this function. Remember this is what the function looks like:

status = lc_set_attr(job, key, (LM_A_VAL_TYPE)value)

Now searching the FlexLM documentation I've found the following key:

LM_A_CHECKOUTFILTER
Type
Pointer to a function returning int.
The checkout filter allows you to examine the FEATURE line which is going to be used in an lc_checkout() request, and either allow the checkout to proceed or reject this particular FEATURE line. This filter function will be called with a pointer to the CONFIG struct which is about to be checked out. If this function returns 0, then checkout proceeds, otherwise if this function returns a non-zero value, then the checkout proceeds to the next available FEATURE line. If this function returns a non-zero value and sets the error obtainable from lc_get_errno(), then this value will be the return of lc_checkout(), otherwise, if lc_get_errno() is set to 0 by this function, the result of lc_checkout() would be LM_LOCALFILTER (assuming the checkout was not attempted on further FEATURE lines, or that another FEATURE line did not produce a LM_MAXUSERS/LM_USERSQUEUED result).
Default
None.

Ok, this looks like the one we want. Looking in lm_attr.h we find this:

#define LM_A_CHECKOUTFILTER    45    /* Vendor-defined checkout filter */
                    /* PTR to func returning int */

So this means that a 45 (0x2d hex) will be pushed right before a call to lc_set_attr to define a vendor defined checkout filter. Disassembling our target we find the following:

:0040104D 68B0114000 	push 004011B0 <-- value = 004011b0 = Address of checkout filter routine

:00401052 6A2D 		push 0000002D <-- LM_A_CHECKOUTFILTER

:00401054 50 		push eax



* Reference To: LMGR326A.lc_set_attr, Ord:003Dh

|

:00401055 E8C8010000 	Call 00401222 <-- Call function

So now we know that the vendor checkout filter routine is at address 0x004011b0. We also know that this routine returns a 0 if everything is OK a non-zero if not OK. This routine will basically check the vendor defined string on the license file line for the feature to check out. Let's take a look at this routine.

.data:00406120 50 6C 61 74 66 6F+aPlatformNt db 'Platform:NT',0 ; Correct vendor string = "Platform:NT"

                                   ; DATA XREF: .text:004011B9o 



.text:004011B0 		vendor_chkout_filter: ; DATA XREF: _main+4Do

.text:004011B0 8B 44 24 04 	mov eax, [esp+4]

.text:004011B4 53 		push ebx

.text:004011B5 56 		push esi

.text:004011B6 8B 70 74 	mov esi, [eax+74h] ; vendor string from license file

.text:004011B9 B8 20 61 40 00 	mov eax, offset aPlatformNt ; correct vendor string

.text:004011BE 		loc_0_4011BE: ; CODE XREF: .text:004011E0j

.text:004011BE 8A 10 		mov dl, [eax] ; move a char of correct string to dl

.text:004011C0 8A 1E 		mov bl, [esi] ; move a char of license file string to bl

.text:004011C2 8A CA 		mov cl, dl ; save copy of correct char in cl

.text:004011C4 3A D3 		cmp dl, bl ; compare char of correct string to char of license file string

.text:004011C6 75 28 		jnz short exit_error ; get out if chars don't match

.text:004011C8 84 C9 		test cl, cl ; test to see if we're at the end of the string

.text:004011CA 74 16 		jz short exit_success ; if end of string then jump to sucess exit

.text:004011CC 8A 50 01 	mov dl, [eax+1] ; move next char of correct string to dl

.text:004011CF 8A 5E 01 	mov bl, [esi+1] ; move next char of license string to bl

.text:004011D2 8A CA 		mov cl, dl ; save copy of correct char in cl

.text:004011D4 3A D3 		cmp dl, bl ; compare char of correct string to char of license file string

.text:004011D6 75 18 		jnz short exit_error ; get out if chars don't match

.text:004011D8 83 C0 02 	add eax, 2 ; increment into string by 2

.text:004011DB 83 C6 02 	add esi, 2 ; increment into string by 2

.text:004011DE 84 C9 		test cl, cl ; test to see if we're at the end of the string

.text:004011E0 75 DC 		jnz short loc_0_4011BE ; jump back if more string to process

.text:004011E2 		exit_success: ; CODE XREF: .text:004011CAj

.text:004011E2 33 C0 		xor eax, eax ; success returns 0 in eax

.text:004011E4 33 C9 		xor ecx, ecx

.text:004011E6 85 C0 		test eax, eax

.text:004011E8 0F 95 C1 	setnz cl

.text:004011EB 8B C1 		mov eax, ecx

.text:004011ED 5E 		pop esi

.text:004011EE 5B 		pop ebx

.text:004011EF C3 		retn 

.text:004011F0 		exit_error: ; CODE XREF: .text:004011C6j

.text:004011F0 1B C0 			; .text:004011D6j

.text:004011F0 			sbb eax, eax ; failure returns non zero in eax

.text:004011F2 5E 		pop esi

.text:004011F3 83 D8 FF 	sbb eax, 0FFFFFFFFh

.text:004011F6 33 C9 		xor ecx, ecx

.text:004011F8 85 C0 		test eax, eax

.text:004011FA 0F 95 C1 	setnz cl

.text:004011FD 8B C1 		mov eax, ecx

.text:004011FF 5B 		pop ebx

.text:00401200 		unknown_libname_1:

.text:00401200 C3 		retn

I've commented the above function and renamed some things using IDA. This helps somewhat to follow the flow of execution. As you can see the routine I've impemented as a vendor defined checkout filter simply checks for a fixed string. A vendor will probably use a more complicated routine and may be worth trying to follow so that a correct license file line can be generated. This example is pretty simple to follow and a real license should be able to be generated. Normally a license line will look like this:

FEATURE f1 blenderd 1.0 permanent uncounted 018FE3AA5EE4 HOSTID=ANY

But with a vendor defined checkout filter it must look like this:

FEATURE f1 blenderd 1.0 permanent uncounted 018FE3AA5EE4 VENDOR_STRING=Platform:NT HOSTID=ANY

The VENDOR_STRING has a value of Platform:NT, which is tested for in my routine. This value is encoded in the encryption key and cannot be changed without invalidating the key.

For those targets that use an extremely complex routine that can't be easily figured out you can simply patch the function to always return 0. In this case the checkout filter routine would look something like this:

.text:004011B0 		vendor_chkout_filter: ; DATA XREF: _main+4Do



.text:004011B0 6A 00	 	push 00000000 ; Push 0 onto stack

.text:004011B4 58 		pop eax ; pop it right off into eax

.text:004011B5 C3 		retn ; return

Below I've included the source code for the target application I've provided. You can see the baselines for imiementing such features into a FlexLM based target.

Here's the code listing of the target vendor_chkout_filter.c:

/* An Example application that uses a vendor defined checkout filter routine */

/* Modified from the SDK version of lmflex.c by: Amante4  1/23/00 */

#include <stdio.h>
#if (defined( __STDC__) || defined(_WINDOWS)) && !defined(apollo)
#include <stdlib.h>
#endif
#include <time.h>
#include "lmclient.h"
#include "lm_code.h"
#include "lm_attr.h"
#define LICPATH "license.dat;."

#define FEATURE "f1"
LM_CODE(code, ENCRYPTION_SEED1, ENCRYPTION_SEED2, VENDOR_KEY1,
    VENDOR_KEY2, VENDOR_KEY3, VENDOR_KEY4, VENDOR_KEY5);

main()
{
char feature[MAX_FEATURE_LEN+1];
LM_HANDLE *lm_job;

    int my_filter( CONFIG *); // prototype for checkout filter routine


    if (lc_init((LM_HANDLE *)0, VENDOR_NAME, &code, &lm_job))
    {
        lc_perror(lm_job, "lc_init failed");
        exit(lc_get_errno(lm_job));
    }

    lc_set_attr(lm_job, LM_A_CHECKOUTFILTER,(LM_A_VAL_TYPE)my_filter); // Tell FlexLM I'm using a checkout filter

    printf("Enter feature to checkout [default: \"%s\"]: ", FEATURE);

    fgets(feature, MAX_FEATURE_LEN, stdin);
    feature[strlen(feature)-1] = '\0'; /* truncate trailing newline */
    if (!*feature) strcpy(feature, FEATURE);


    lc_set_attr(lm_job, LM_A_LICENSE_DEFAULT, (LM_A_VAL_TYPE)LICPATH);

    if (lc_checkout(lm_job, feature, "1.0", 1, LM_CO_NOWAIT, &code,
                                LM_DUP_NONE))
    {
        printf("checkout failed... press return to exit...\n");
        getchar();
        lc_perror(lm_job, "checkout failed");
        lc_free_job(lm_job);
        exit(lc_get_errno(lm_job));
    }

    printf("%s checked out...press return to exit...", feature);
/*
*    Wait till user hits return
*    getchar may be interrupted by SIGALRM, so we loop if necessary
*/
    getchar();
    lc_checkin(lm_job, feature ,0);

    lc_free_job(lm_job);
    return(0);
}

/* My vendor Checkout Filter routine */
int my_filter( CONFIG *input)
{
    char *vendor_string;
    vendor_string = input->lc_vendor_def; // get the vendor defined string from the structure (see config structure in lmclient.h)
    if (strcmp("Platform:NT", vendor_string)) { // see if it matches what we expect
        return (1); // fail if not equal
    }
    else {
        return (0); // success because the vendor string is OK
    }
}

Once again FlexLM proves to be easily defeated. The customization mentioned in this essay prevents a generic crack of FlexLM if the vendor impements it. This will not stop a dedicated cracker

from busting the protection as seen above. There's still some more FlexLM work to be done. Look for other essays to follow from me on this subject.

Shouts out to Volatility, S^witz, VisionZ-, alpine, Muad'Dib, aLoNg3x, Torn@do, all the other ID members, jpk, and anyone else I've missed (probably quite a few;))

later, Amante4

Remember that you can do all the things written here only at YOUR risk.
I do NOT take any liability for YOUR acts.