Q:

How to do Math(subtraction, multiplication, factorial) and change output to hexadecimal in lc3?

I'm using a LC3 simulator and I want to change the output from decimal to hexadecimal, and as I said in the title I'm wondering how could I do Math operations in LC3. I'm very new to LC3 so I really need help.Thanks.
;
; Initialization
;
        .ORIG   x3000
        LD      R6, EMPTY   ; R6 is the stack pointer
        LD      R5, PTR     ; R5 is pointer to characters
        AND     R0, R0, #0
        ADD     R0, R0, #10 ; Print a new line
        OUT
;   
REDO    LDR     R3, R5, #0  ; R3 gets character
;
; Test character for end of file
;       
        ADD     R4, R3, #-10    ; Test for end of line (ASCII xA)
        BRz     EXIT        ; If done, quit
        LD      R4, ZERO
        ADD     R3, R3, R4  ; Get the decimal value from ASCII
        JSR     CONV
        ADD     R5, R5, #1
        AND     R4, R5, #1  ; check odd/even
        BRz     EVEN
        ADD     R2, R3, #0  ; Save the first operand to R2
        LD      R0, PLUS    ; '+'
        OUT
        BRnzp   REDO
EVEN    LD      R0, EQUAL       ; '='
        OUT
; Start calculation
        ADD     R3, R2, R3  ; The second operand is at R3
;
        JSR     CONV
        AND     R0, R0, #0
        ADD     R0, R0, #10 ; Print a new line
        OUT
        BRnzp   REDO        
;
; A subroutine to output a 3-digit decimal result.
;
CONV    ADD     R1, R7, #0  ; R3, R4, R5 and R7 are used in this subroutine
        JSR     Push
        ADD     R1, R3, #0  ; R3 is the input value
        JSR     Push
        ADD     R1, R4, #0
        JSR     Push
        ADD     R1, R5, #0
        JSR     Push
        AND     R5, R5, #0
OUT100  LD      R4, HUNDRED
        ADD     R4, R3, R4  ; R3 - #100
        BRn     PRI100
        LD      R4, HUNDRED
        ADD     R3, R3, R4  ; R3 - #100
        ADD     R5, R5, #1
        BRnzp   OUT100
PRI100  LD      R0, ASCII   ; Load the ASCII template
        ADD     R0, R0, R5      ; Convert binary count to ASCII
        OUT                     ; ASCII code in R0 is displayed.
        AND     R5, R5, #0
OUT10   ADD     R4, R3, #-10
        BRn     PRI10
        ADD     R3, R3, #-10
        ADD     R5, R5, #1
        BRnzp   OUT10
PRI10   LD      R0, ASCII       ; Load the ASCII template
        ADD     R0, R0, R5  ; Convert binary count to ASCII
        OUT                   ; ASCII code in R0 is displayed.      
        LD      R0, ASCII
        ADD     R0, R0, R3      ; Convert binary count to ASCII
        OUT                     ; ASCII code in R0 is displayed.
        JSR     Pop
        ADD     R5, R1, #0
        JSR     Pop
        ADD     R4, R1, #0
        JSR     Pop
        ADD     R3, R1, #0
        JSR     Pop
        ADD     R7, R1, #0
        RET
Push    STR     R1, R6, #0      ; Stack Push
        ADD     R6, R6, #-1 
        RET 
Pop     ADD     R6, R6, #1      ; Stack Pop
        LDR     R1, R6, #0
        RET
; End of the subroutine
EXIT    HALT                    ; Halt machine

PTR     .FILL   x3500
EMPTY   .FILL   x4000 
ASCII   .FILL   x0030           ; '0'
ZERO    .FILL   xFFD0           ; -'0'
HUNDRED .FILL   xFF00           ; -x100
EQUAL   .FILL   x003D           ; '='
PLUS    .FILL   x002B           ; '+'
MINUS   .FILL   x002D           ; '-'
FACTOR  .FILL   x0021           ; '!'
MULT    .FILL   x002A           ; '*'
CHAR_A  .FILL   x0041           ; 'A'
VAL     .BLKW   1
        .END

I get the inputs from another file and they are like this, so on the simulator it'll show like '002+003=005' for now. And I want it to show '009+008=011', '009-008=001', '009*008=048', '006!=2D0' for addition, subtraction, multiplication and factorial.
.ORIG       x3500
HELLO   .STRINGZ    "9887766554433221\n" ;this is the other file

A:

How do I do subtraction?

The LC3 has no subtraction instruction. What you need to do is take the negative of the second operand, and add it to the first operand. How do you negate the second operand? The LC3 is a Two's Complement machine. So, you do a bitwise not, then add 1.
To add two numbers in R0 and R1, placing the result in R0, use the following assembly:
ADD R0, R0, R1

To subtract R1 from R0, (i.e. R0 - R1) and place the result in R0, use the following assembly:
NOT R1, R1
ADD R1, R1, #1
ADD R0, R0, R1

That was a lot of work, but at least it's not multiplication!

How do I do multiplication?

...
Fine.
There are two ways to approach multiplication. The first is repeated addition. Imagine you are multiplying 5 by 4.
 5
 5
 5
+5
--
20

So, you write a loop, and repeatedly add 5, 4 times. This gets unwieldy fast if you're trying to multiply large numbers.
The second way is to use a mask, and shift the mask and one of the operands left once each loop. This allows you to do the multiply in only 16 loops.

How do I do factorial?

You create a multiplication subroutine, and you use that repeatedly to create a factorial subroutine.

I want to change the output from decimal to hexadecimal

You're in luck, because hexadecimal is much easier to deal with in the LC3 than decimal. What you can do is take the top 4 bits of the number, shift them right, and use a lookup table to decide what hex digit to print. Then, shift your original number left by 4 bits. Do this 4 times, and you'll have printed out the number.
Here's an example:
; assume R0 has the 4-bit number we want to convert to hex
LEA R1, lookup
ADD R0, R0, R1 ; go to the cell pointed to by R1, and advance the pointer
               ; by the number in R1
LDR R0, R0, #0 ; load the cell pointed to by R0 into R0
OUT            ; print hex
lookup .STRINGZ "0123456789ABCDEF"

Shifting right can be annoying, depending on what version of the LC3 you use. If you use LC3b, you have a shift right instruction built in. Otherwise, you'll need to write a loop.

