C struct program crash

Question 1

Try these simple changes to your original code: (builds and runs without using malloc() or calloc())

(comments in-line to explain)

#include <stdio.h>
 #include <stdlib.h>
 typedef struct{
         int x;
         int y;

        }structure; 
typedef struct{
        structure test1;
        }TEST; //use CAPITALS to distinguish typedefed struct for readability

TEST test;  //use typedef to create instance of TEST

void function(TEST *trying){
      trying->test1.x = 5; 

     printf("%d\n", trying->test1.x);

     }

int main(){
   TEST *mystruct; // create local instance of TEST

   mystruct = &test;//initialize mystruct pointer to point to beginning of physical struct
   function(mystruct);
   system("pause");
   return 0;
}

Question 2

test *mystruct;
function(mystruct);

mystruct pointer is not initialized and has an indeterminate value.

This statement invokes undefined behavior because of the absence of initialization:

trying->test1.x = 5;

Do that instead:

test mystruct;
function(&mystruct);

Question 3

When you write test *mystruct;, mystruct contains garbage value. If this value lies outside the address space for your program, you'll get the error. Why crash? Because it tries to overwrite protected areas of memory like the OS. So either you allocate memory for mystruct and use it (You'll have to free the memory in this case). Or plain declare as normal variable and pass the address to the function.

test mystruct;
function(&mystruct);

Question 4

In function main, variable mystruct is not initialized to point to a valid memory address. Therefore, it is most likely pointing to an invalid memory address, and when used in function function, it eventually yields a memory access violation.

Fix suggestion #1:

test mystruct;
function(&mystruct);

Fix suggestion #2:

test* mystruct = malloc(sizeof(test));
function(mystruct);
free(mystruct);

Question 5

if you don't want to use dynamic memory then try this:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

typedef struct{
     int x;
     int y;
}structure; 
typedef struct{
    structure test1;
}test;

void function(test *trying){
    trying->test1.x = 5; 
    printf("%d\n", trying->test1.x);
}

int main(){
   test mystruct;
   memset(&mystruct, 0, sizeof(mystruct));
   function(&mystruct);
   system("pause");
   return 0;
}

This creates a stack based variable and initializes its contents using the memset() function. You have an odd intermediate typedef and member that you might eliminate entirely. e.g.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

typedef struct my_s{
     int x;
     int y;
} my_t;

void function(my_t *t){
   if(!t) return;
   t->x = 5;
   printf("%d\n", t->x);
}

int main(){
   my_t mystruct;
   memset(&mystruct, 0, sizeof(mystruct));
   function(&mystruct);
   return 0;
}

Lastly, Im not sure what you're trying to accomplish with the system("pause") but as for the structure, the above examples will work.

Question 6

You can use the second half of this answer but as other commenters have noted, it's much better to simply pass the address of mystruct to the function.

test mystruct;     // = (test *) malloc(sizeof(test));
function(&mystruct);

This means it's allocated on the stack, which is different than the heap, and you don't have to free the memory after you use it because it's done for you.

You can to allocate memory on the heap for test if you're going to declare it as a pointer.

Change your first line in main to:

test *mystruct = malloc(sizeof(test));

And free the memory that was allocated after you're done using it:

free(mystruct);